sonic  scenery  :  interactive  storytelling  through  sound  and  music   MMus  thesis  proposal  for  Music  Technology  Innovation,  Berklee  College  of  Music  2014   Beth  Michelle  Schofield                                 Sonic   Scenery   is   a   project   exploring   interactive   storytelling   through   sound.   I   am   producing  a  hybrid  installation-­‐live  performance  piece  that  uses  music  and  technology   to  take  listeners  on  a  journey  through  emotion  from  different  points  of  view.  Thanks  to   an  invitation  to  join  the  Walk  to  Fisterra  team,  I  took  on  the  Camino  de  Santiago  where  I   was   able   to   encourage   other   ‘pilgrims’   to   share   their   experiences.   I   have   produced   a   3   movement  audiovisual  work  with  the  possibility  of  live  performance.     The  following  report  documents  the  experience.     Contents     Introduction                 3             4           5             6     Description  of  the  Work     Innovative  aspects  of  the  Work       New  Skills  Acquired       Challenges                 7   The  future                 8   Conclusion                 8                                 2   Introduction       The   work   I   am   presenting   is   the   product   of   a   journey   –   educational,   spiritual   and   literal.   I   have   developed   in   many   ways   this   year   and   with   an   onslaught   of   new   knowledge,   comes   a   terrifying   but   wonderful   sense   of   understanding   so   much   more   but   not   yet   knowing   how   to   use   this   knowledge.     I   flirted   with   as   many   different   possible   new   skills   as  I  could  this  year  hoping  to  integrate  them  all.    My  plans  were  too  extensive,  for  this   year   at   least,   and   I   was   getting   concerned   of   the   risk   of   not   actually   producing   anything.   At   the   lowest   point,   an   answer   came   in   the   form   of   an   invitation   to   join   a   film-­‐crew   making   a   documentary   –   an   idea   I   had   not   seriously   entertained   since   my   young   self   came  to  the  realization  that  I  could  never  be  Michael  Palin.     I   joined   them   as   a   sound   engineer   along   with   Kyle   Pyke.   We   brought   a   lot   of   very   expensive,   sought   after   audio   equipment,   and   as   a   team   we   walked   the   Camino   de   Santiago.  Whilst  the  project  started  as  a  documentation  of  Dane  Johansen’s  travels  with   his  cello  and  recording  sessions  in  the  churches  along  the  way,  it  morphed  into  a  much   more  selfless  offering  to  the  camino  community,  looking  at  the  stories  of  many  pilgrims   and  instead  of  private  recording  sessions,  offering  free  concerts.     Every  day  we  had  two  Units  –  Van  unit  and  Walking  Unit,  seeking  out  the  best  nuggets  of   the  camino  to  capture.  I  cam  away  with  a  plethora  of  raw  footage  and  have  made  it  into  a   3   movement   suite   with   visuals   to   express   the   emotional   journey   of   some   of   the   pilgrims   we  met  along  the  way.             3   Description  of  the  Work     The  work  is  a  through-­‐composed  piece  in  three  parts  for  audio  and  video.     The   first   movement   is   made   up   entirely   of   field   recordings   from   the   camino.   The   recorder   melody   was   recorded   at   the   church   of   Zabaldika   with   a   shotgun   up   in   the   gallery  to  emulate  the  sound  of  standing  above  looking  down  over  the  player.     I  had  already  programmed  the  second  movement  as  a  standalone  piece  in  Max  MSP  so   the   final   element   for   that   was   trawling   through   the   audio   footage   to   extract   some   moments  I  found  particularly  expressed  the  emotion  I  was  portraying  in  my  lyrics.     The   final   part   is   an   arrangement   of   Feliciano’s   setting   of   R.   Tagore’s   poem   Silence   My   Soul.  Before  I  left  for  the  camino  I  felt  like  this  would  be  a  perfect  summing  up  and  when   I  reached  Finisterre,   this  was  confirmed.  The  natural  sounds   in   this  movement  are  from   Finisterre   and   the   spoken   word   was   recorded   in   the   Berklee   studios   on   our   return.   I   originally  planned  to  find  all  the  words  of  the  poem  in  interviews  from  the  camino  but   quickly   realised   I   did   not   have   time   for   that   and   there   was   no   reason   to   only   use   pilgrim   interviews  when  really  the  piece  is  about  an  emotional  cycle  we  all  experience  whether   on  a  camino  or  not.     The  second  and  third  movements  can  both  be  performed  live  with  Max  MSP  with  a  live   performer   taking   the   primary   role.   The   versions   in   the   film   I   am   presenting   feature     recordings   of   Ganavya   Doraiswamy   singing   in   Mvmt   2   and   myself   playing   recorder   in   Mvmt   3,   both   live   with   the   program   recorded   directly   from   Max.   I   added   the   field   recordings   to   Mvmt   2   after   the   live   recording   but   Mvmt   3   is   entirely   recorded   in   Max.   The  additions  to  Mvmt  2  and  the  whole  1st  movement  were  edited  in  ProTools  and  the   final  mix  and  master  was  also  in  ProTools.     4   Innovative  aspects  of  the  work     The   concept   of   presenting   a   work   in   a   new   form   is   something   that   has   caught   my   imagination.  With  an  initial  proposal  to  produce  an  interactive  promenade  theatre  piece,   I   started   to   learn   to   program   in   Max   MSP   and   Jitter   and   designed   the   system   to   run   such   a  show.  From  that,  I  was  able  to  experiment  with  new  ways  to  allow  anyone  to  perform   with  technology  without  requiring  specialist  knowledge.  The  two  parts  of  my  work  that   can   be   performed   live   with   Max,   whilst   complex   under   the   surface,   have   friendly   user   interfaces  allowing  anyone  to  perform  with  the  piece.  This  was  proven  at  an  event  at  the   Museo   de   las   Bellas   Artes   in   Valencia   where   Ganavya   Doraiswamy   performed   the   2nd   Mvmt.   With   no   prior   Max   experience   she   was   able   to   perform   a   complex   multichannel   piece   controlling   the   software   with   the   tool   she   knows   best   –   her   voice.   When   performed  live,  the  3rd  Mvmt  will  be  such  that  any  audience  member  can  take  the  role  of   the  main  orator  and  the  piece  will  listen  to  them.     For   me,   this   is   the   beginnings   of   an   understanding   and   implementation   of   true   interactivity  and  immersion.                         5   New  Skills  Acquired     This  year  has  given  me  more  new  skills  than  I  could  have  hoped  for.  I  now  feel  confident   in  a  myriad  of  areas  all  of  which  have  been  of  immense  use  whilst  making  this  work.   -­‐ Programming  in  Max  MSP/Jitter     This  has  been  an  invaluable  skills  which  I  did  not  anticipate  learning  or   enjoying   as   thoroughly   as   I   have.   There   is   a   long   way   to   go   but   I   feel   that   every   day   I   become   more   comfortable   in   this   medium   and   I   love   to   compose  and  design  with  it.   -­‐ Recording  and  Mixing  skills  on  large  format  consoles   After   a   year   of   watching,   I   am   astounded   how   comfortable   I   know   feel   working  on  the  AKSS.   -­‐ Audio  editing   I  am  much  quicker  and  more  confident  now.   -­‐ Video  editing   Another  skill  I  did  not  anticipate  learning  at  all.  I  would  like  to  continue   developing  this  to  improve  my  personal  brand  image.   -­‐ On-­‐location  film  audio  techniques  and  etiquette   Life   as   a   sound   engineer   on   a   film-­‐crew   is   in   many   ways   very   different   to   that  in  the  studio.  It  is  true  it  is  sometimes  overlooked  with  precedence   given   to   camera   but   in   reality   it   is   the   unit   dynamics   that   dictate   the   quality  of  any  footage  captured,  camera  or  sound.   -­‐ Troubleshooting  and  patience   Probably  the  most  useful  skill  I  have  developed.   -­‐ Be  open  to  change  and  do  not  limit  yourself  with  your  own  expectations   Probably  the  joint  most  important  thing  I  have  learned.       6   Challenges     Time   has   been   a   recurring   challenge   for   this   year   and   this   filming   project.   We   were   constantly   pushing   to   get   to   the   next   venue,   set   up   before   a   Missa,   capture   interviews   before  our  subjects  moved  on  to  the  next  pueblo.  In  many  ways,  on-­‐location  recording,   in   particular   the   Camino   de   Santiago   environment,   is   the   polar   opposite   of   that   as   fostered  on  a  film  set  or  in  a  recording  studio.    Very  little  can  be  planned  and  anything   you  do  plan  rarely  turns  out  as  expected.  We  all  had  to  accept  that  any  expectations  we   had  were  limiting  the  experience  and  once  we  let  go  of  those  expectations,  we  started  to   find  some  brilliant  situations.     I  knew  that  time  would  be  an  issue  for  me  personally  all  year  but  hoped  to  improve.  In   reality  I  am  still  frequently  up  against  the  clock  but  I  have  also  learnt  that  the  problem   only   comes   when   the   panic   and   react   adversely   to   this   instead   of   taking   the   opportunity   to   use   that   adrenalin   to   fuel   work.   Thus   giving   an   example   of   limitation   fostering   creativity.     Lack  of  experience  in  film  audio  was  a  challenge.  I  arrived  certain  that  I  could  apply  my   studio  etiquette  and  techniques  and  whilst  true  to  some  extent,  I  found  I  had  to  learn  a   whole  new  vocabulary,  style  of  working,  people  skills  to  succeed.  I  particularly  enjoyed   the  challenge  of  working  with  people  we  met  and  documenting  their  reality  rather  than   the  ‘artist’  in  a  studio  documenting  their  art.     The   biggest   challenge   has   been   self-­‐belief,   whilst   learning   so   many   new   skills,   it   was   easy   to   conclude   I   simply   couldn’t   do   anything   very   well.   Accepting   that   with   learning   new  skills  comes  the  responsibility  of  understanding  that  you  have  a  lot  more  to  learn   has  been  a  great  challenge  to  overcome.     7   The  future  and  further  plans  for  the  work     I  hope  to  take  all  the  skills  I  have  learned  this  year  and  continue  to  design  and  program   innovative   uses   of   technology   to   both   enhance   reality   and   encourage   real   interactivity   and  immersion  for  anyone  without  requiring  technical  knowledge.     The   first   element   to   complete   after   graduation   will   be   my   full   intention   for   the   3rd   movement.   It   will   have   the   ability   to   become   heavily   interactive   allowing   the   guest   to   take   the   role   of   the   primary   orator.   The   programming   of   this   movement   is   such   that   anyone   can   stand   in   front   of   it,   offer   their   musical   ideas   and   their   ‘performance   partner,’   Max   MSP,   will   respond   and   interact   in   a   sensitive   way.   For   this   reason   I   will   be   able   to   also   setup   this   movement   as   a   stand-­‐alone  installation  piece  to  see  if  people  enjoy  interacting  with  it.  This  observation  will   be  extremely  useful  in  development  of  future  interactive  works  also.     The  next  stage  of  development  for  this  piece  –  and  an  exploration  for  future  works  using   similar   technology   –   would   be   to   give   the   audience   more   control   over   the   dialogue   narrative   and   let   them   choose   their   own   path.     Each   audience   member   would   be   given   a   ‘passport’  which  can  ‘read’  the  objects  and  provide  personal  narrative  via  headphones.   The  passport  would  hold  an  app  to  gather  audience  decision-­‐making,  so  I  must  have  a   way   to   send   that   information   from   the   interactive   physical   element   to   the   app.   I   can   achieve  this  using  a  Raspberry,  a  RPi-­‐Arduino  shield  bridge  and  the  Arduino  NFC  shield.       This  shield  can  read  NFC  tags  on  the  objects,  send  information  to  Max  via  Maxuino  and   thus  trigger  the  appropriate  audio  file.  An  extension  would  be  to  run  something  akin  to   wordclock   over   a   wifi   network   to   each   ‘passport’,   which   could   allow   more   specific   control  over  integration  of  the  headphone  audio  with  the  speaker  audio.  Adding  a  WiFI   USB  dongle  to  the  Raspberry  Pi  and  writing  the  patch  to  have  an  overall  clock  keeping     8   an  eye  on  the  piece  and  communicating  musical  time  to  each  device  would  achieve  this.     An  alternative  to  the  Raspberry  Pi  would  be  to  use  an  Arduino  Yun  board  although  more   research  is  needed  to  define  which  would  ultimately  be  more  suitable  for  the  task.     I   would   have   two   options   for   headphones.   One   is   to   use   bone-­‐conduction   technology   for   the  full  effect  of  both  audio  sources  being  experienced  as  intended  without  interference   from  having  headphones  over  ears.  I  have  successfully  made  a  prototype  of  simple  bone-­‐ conducting  headphones  made  from  piezo  transducers.  They  function  well  as  a  prototype   although   the   sound   quality   is   somewhat   low   fidelity   and   I   will   look   into   the   best   amplifier   to   use,   as   the   one   I   used   in   the   prototype   is   not   powerful   enough.   I   could   instead  provide  users  with  an  alternative  solution,  possibly  being  open-­‐backed  earbuds   or   just   one   earpiece.   Whichever   my   decision,   I   also   must   decide   if   the   headphone   feed   will   be   mono   or   stereo.   The   dialogue   will   be   recorded   using   my   own   binaural   in-­‐ear   microphones  so  stereo  would  be  ideal  but  perhaps  overkill  realistically  speaking.     The   next   project   I   would   like   to   work   on   using   the   technology   I   learn   through   this   study   would  be  trying  the  same  ideas  but  with  pre-­‐existing  stories  and  fairytales.  I  would  like   to   focus   on   bringing   together   traditional   theatre   elements   and   new   interaction   technology  ideas  to  provider  hyper-­‐immersive  interactive  theatre  that  the  audience  can   take   true   ownership   in.   This   was   a   project   too   extensive   for   this   year   and   will   require   me  to  find  some  likeminded  people  who  can  offer  their  specialist  skills  to  help  out  in  the   realization.   Through   The   Walk   to   Fisterra   project   I   have   started   to   network   internationally  and  I  am  starting  to  see  that  it  is  a  possibility  to  achieve  this.             9   Conclusion     I   had   no   idea   the   joy   that   I   would   experience   in   capturing   the   emotions   of   others   in   audio.   Spoken   word   as   music   has   always   been   a   interesting   concept   to   me,   sparked   originally  by  Steve  Reich’s  Different  Trains  and  I  now  realise  that  it  is  not  just  a  simple   interest  but  I  really  love  to  listen  and  work  with  spoken  word.     I  also  have  a  new  appreciation  for  the  impact  of  visual  media  and  I  will  be  making  efforts   to   further   my   familiarity   with   this   medium.   I   will   be   undertaking   an   internship   in   the   audio   department   of   a   post-­‐production   house   in   New   York   this   Fall   which   I   anticipate   being  a  very  enlightening  experience  which  I  can  use  to  not  only  work  with  other  people   but  also  implement  in  my  own  work.     In   conclusion,   my   primary   findings   from   this   year   of   study   have   been   confirming   my   idea   that  technology  can  be  used  discretely  and  simply  to  enhance  one’s  experience  of   the   world   around   you.   Now   I   hope   to   continue   bringing   the   audience   further   into   that   world  with  fewer  barriers  to  their  full  experience.                         10