sonic
 scenery
 :
 interactive
 storytelling
 through
 sound
 and
 music
 
MMus
 thesis
 proposal
 for
 Music
 Technology
 Innovation,
 Berklee
 College
 of
 Music
 2014
 
Beth
 Michelle
 Schofield
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 
Sonic
  Scenery
  is
  a
  project
  exploring
  interactive
  storytelling
  through
  sound.
  I
  am
 
producing
 a
 hybrid
 installation-­‐live
 performance
 piece
 that
 uses
 music
 and
 technology
 
to
 take
 listeners
 on
 a
 journey
 through
 emotion
 from
 different
 points
 of
 view.
 Thanks
 to
 
an
 invitation
 to
 join
 the
 Walk
 to
 Fisterra
 team,
 I
 took
 on
 the
 Camino
 de
 Santiago
 where
 I
 
was
  able
  to
  encourage
  other
  ‘pilgrims’
  to
  share
  their
  experiences.
  I
  have
  produced
  a
  3
 
movement
 audiovisual
 work
 with
 the
 possibility
 of
 live
 performance.
 
 
The
 following
 report
 documents
 the
 experience.
 

 

Contents
 

 
Introduction
 
 


 


 


 


 


 


 

3
 


 


 


 


 


 

4
 


 


 


 


 

5
 


 


 


 


 


 

6
 


 
Description
 of
 the
 Work
 

 
Innovative
 aspects
 of
 the
 Work
 

 


 

New
 Skills
 Acquired
 
 

 
Challenges
 


 


 


 


 


 


 


 

7
 

The
 future
 


 


 


 


 


 


 


 

8
 

Conclusion
 


 


 


 


 


 


 


 

8
 


 


 


 

 

 

 

 

 

 

 

 

 

 

 


 

2
 

Introduction
 
 

 
The
  work
  I
  am
  presenting
  is
  the
  product
  of
  a
  journey
  –
  educational,
  spiritual
  and
  literal.
 
I
  have
  developed
  in
  many
  ways
  this
  year
  and
  with
  an
  onslaught
  of
  new
  knowledge,
 
comes
  a
  terrifying
  but
  wonderful
  sense
  of
  understanding
  so
  much
  more
  but
  not
  yet
 
knowing
  how
  to
  use
  this
  knowledge.
 
  I
  flirted
  with
  as
  many
  different
  possible
  new
  skills
 
as
 I
 could
 this
 year
 hoping
 to
 integrate
 them
 all.
 
 My
 plans
 were
 too
 extensive,
 for
 this
 
year
  at
  least,
  and
  I
  was
  getting
  concerned
  of
  the
  risk
  of
  not
  actually
  producing
  anything.
 
At
  the
  lowest
  point,
  an
  answer
  came
  in
  the
  form
  of
  an
  invitation
  to
  join
  a
  film-­‐crew
 
making
  a
  documentary
  –
  an
  idea
  I
  had
  not
  seriously
  entertained
  since
  my
  young
  self
 
came
 to
 the
 realization
 that
 I
 could
 never
 be
 Michael
 Palin.
 

 
I
  joined
  them
  as
  a
  sound
  engineer
  along
  with
  Kyle
  Pyke.
  We
  brought
  a
  lot
  of
  very
 
expensive,
  sought
  after
  audio
  equipment,
  and
  as
  a
  team
  we
  walked
  the
  Camino
  de
 
Santiago.
 Whilst
 the
 project
 started
 as
 a
 documentation
 of
 Dane
 Johansen’s
 travels
 with
 
his
 cello
 and
 recording
 sessions
 in
 the
 churches
 along
 the
 way,
 it
 morphed
 into
 a
 much
 
more
 selfless
 offering
 to
 the
 camino
 community,
 looking
 at
 the
 stories
 of
 many
 pilgrims
 
and
 instead
 of
 private
 recording
 sessions,
 offering
 free
 concerts.
 

 
Every
 day
 we
 had
 two
 Units
 –
 Van
 unit
 and
 Walking
 Unit,
 seeking
 out
 the
 best
 nuggets
 of
 
the
 camino
 to
 capture.
 I
 cam
 away
 with
 a
 plethora
 of
 raw
 footage
 and
 have
 made
 it
 into
 a
 
3
  movement
  suite
  with
  visuals
  to
  express
  the
  emotional
  journey
  of
  some
  of
  the
  pilgrims
 
we
 met
 along
 the
 way.
 

 

 

 

 


 

3
 

Description
 of
 the
 Work
 

 
The
 work
 is
 a
 through-­‐composed
 piece
 in
 three
 parts
 for
 audio
 and
 video.
 
 
The
  first
  movement
  is
  made
  up
  entirely
  of
  field
  recordings
  from
  the
  camino.
  The
 
recorder
  melody
  was
  recorded
  at
  the
  church
  of
  Zabaldika
  with
  a
  shotgun
  up
  in
  the
 
gallery
 to
 emulate
 the
 sound
 of
 standing
 above
 looking
 down
 over
 the
 player.
 

 
I
 had
 already
 programmed
 the
 second
 movement
 as
 a
 standalone
 piece
 in
 Max
 MSP
 so
 
the
  final
  element
  for
  that
  was
  trawling
  through
  the
  audio
  footage
  to
  extract
  some
 
moments
 I
 found
 particularly
 expressed
 the
 emotion
 I
 was
 portraying
 in
 my
 lyrics.
 

 
The
  final
  part
  is
  an
  arrangement
  of
  Feliciano’s
  setting
  of
  R.
  Tagore’s
  poem
  Silence
  My
 
Soul.
 Before
 I
 left
 for
 the
 camino
 I
 felt
 like
 this
 would
 be
 a
 perfect
 summing
 up
 and
 when
 
I
 reached
 Finisterre,
  this
 was
 confirmed.
 The
 natural
 sounds
  in
  this
 movement
 are
 from
 
Finisterre
  and
  the
  spoken
  word
  was
  recorded
  in
  the
  Berklee
  studios
  on
  our
  return.
  I
 
originally
 planned
 to
 find
 all
 the
 words
 of
 the
 poem
 in
 interviews
 from
 the
 camino
 but
 
quickly
  realised
  I
  did
  not
  have
  time
  for
  that
  and
  there
  was
  no
  reason
  to
  only
  use
  pilgrim
 
interviews
 when
 really
 the
 piece
 is
 about
 an
 emotional
 cycle
 we
 all
 experience
 whether
 
on
 a
 camino
 or
 not.
 

 
The
 second
 and
 third
 movements
 can
 both
 be
 performed
 live
 with
 Max
 MSP
 with
 a
 live
 
performer
  taking
  the
  primary
  role.
  The
  versions
  in
  the
  film
  I
  am
  presenting
  feature
 
 
recordings
  of
  Ganavya
  Doraiswamy
  singing
  in
  Mvmt
  2
  and
  myself
  playing
  recorder
  in
 
Mvmt
  3,
  both
  live
  with
  the
  program
  recorded
  directly
  from
  Max.
  I
  added
  the
  field
 
recordings
  to
  Mvmt
  2
  after
  the
  live
  recording
  but
  Mvmt
  3
  is
  entirely
  recorded
  in
  Max.
 
The
 additions
 to
 Mvmt
 2
 and
 the
 whole
 1st
 movement
 were
 edited
 in
 ProTools
 and
 the
 
final
 mix
 and
 master
 was
 also
 in
 ProTools.
 


 

4
 

Innovative
 aspects
 of
 the
 work
 

 
The
  concept
  of
  presenting
  a
  work
  in
  a
  new
  form
  is
  something
  that
  has
  caught
  my
 
imagination.
 With
 an
 initial
 proposal
 to
 produce
 an
 interactive
 promenade
 theatre
 piece,
 
I
  started
  to
  learn
  to
  program
  in
  Max
  MSP
  and
  Jitter
  and
  designed
  the
  system
  to
  run
  such
 
a
 show.
 From
 that,
 I
 was
 able
 to
 experiment
 with
 new
 ways
 to
 allow
 anyone
 to
 perform
 
with
 technology
 without
 requiring
 specialist
 knowledge.
 The
 two
 parts
 of
 my
 work
 that
 
can
  be
  performed
  live
  with
  Max,
  whilst
  complex
  under
  the
  surface,
  have
  friendly
  user
 
interfaces
 allowing
 anyone
 to
 perform
 with
 the
 piece.
 This
 was
 proven
 at
 an
 event
 at
 the
 
Museo
  de
  las
  Bellas
  Artes
  in
  Valencia
  where
  Ganavya
  Doraiswamy
  performed
  the
  2nd
 
Mvmt.
  With
  no
  prior
  Max
  experience
  she
  was
  able
  to
  perform
  a
  complex
  multichannel
 
piece
  controlling
  the
  software
  with
  the
  tool
  she
  knows
  best
  –
  her
  voice.
  When
 
performed
 live,
 the
 3rd
 Mvmt
 will
 be
 such
 that
 any
 audience
 member
 can
 take
 the
 role
 of
 
the
 main
 orator
 and
 the
 piece
 will
 listen
 to
 them.
 

 
For
  me,
  this
  is
  the
  beginnings
  of
  an
  understanding
  and
  implementation
  of
  true
 
interactivity
 and
 immersion.
 

 

 

 

 

 

 

 

 

 

 


 

5
 

New
 Skills
 Acquired
 

 
This
 year
 has
 given
 me
 more
 new
 skills
 than
 I
 could
 have
 hoped
 for.
 I
 now
 feel
 confident
 
in
 a
 myriad
 of
 areas
 all
 of
 which
 have
 been
 of
 immense
 use
 whilst
 making
 this
 work.
 
-­‐

Programming
 in
 Max
 MSP/Jitter
 
 
This
 has
 been
 an
 invaluable
 skills
 which
 I
 did
 not
 anticipate
 learning
 or
 
enjoying
  as
  thoroughly
  as
  I
  have.
  There
  is
  a
  long
  way
  to
  go
  but
  I
  feel
  that
 
every
  day
  I
  become
  more
  comfortable
  in
  this
  medium
  and
  I
  love
  to
 
compose
 and
 design
 with
 it.
 

-­‐

Recording
 and
 Mixing
 skills
 on
 large
 format
 consoles
 
After
  a
  year
  of
  watching,
  I
  am
  astounded
  how
  comfortable
  I
  know
  feel
 
working
 on
 the
 AKSS.
 

-­‐

Audio
 editing
 
I
 am
 much
 quicker
 and
 more
 confident
 now.
 

-­‐

Video
 editing
 
Another
 skill
 I
 did
 not
 anticipate
 learning
 at
 all.
 I
 would
 like
 to
 continue
 
developing
 this
 to
 improve
 my
 personal
 brand
 image.
 

-­‐

On-­‐location
 film
 audio
 techniques
 and
 etiquette
 
Life
  as
  a
  sound
  engineer
  on
  a
  film-­‐crew
  is
  in
  many
  ways
  very
  different
  to
 
that
 in
 the
 studio.
 It
 is
 true
 it
 is
 sometimes
 overlooked
 with
 precedence
 
given
  to
  camera
  but
  in
  reality
  it
  is
  the
  unit
  dynamics
  that
  dictate
  the
 
quality
 of
 any
 footage
 captured,
 camera
 or
 sound.
 

-­‐

Troubleshooting
 and
 patience
 
Probably
 the
 most
 useful
 skill
 I
 have
 developed.
 

-­‐

Be
 open
 to
 change
 and
 do
 not
 limit
 yourself
 with
 your
 own
 expectations
 
Probably
 the
 joint
 most
 important
 thing
 I
 have
 learned.
 


 


 

6
 

Challenges
 

 
Time
  has
  been
  a
  recurring
  challenge
  for
  this
  year
  and
  this
  filming
  project.
  We
  were
 
constantly
  pushing
  to
  get
  to
  the
  next
  venue,
  set
  up
  before
  a
  Missa,
  capture
  interviews
 
before
 our
 subjects
 moved
 on
 to
 the
 next
 pueblo.
 In
 many
 ways,
 on-­‐location
 recording,
 
in
  particular
  the
  Camino
  de
  Santiago
  environment,
  is
  the
  polar
  opposite
  of
  that
  as
 
fostered
 on
 a
 film
 set
 or
 in
 a
 recording
 studio.
 
 Very
 little
 can
 be
 planned
 and
 anything
 
you
 do
 plan
 rarely
 turns
 out
 as
 expected.
 We
 all
 had
 to
 accept
 that
 any
 expectations
 we
 
had
 were
 limiting
 the
 experience
 and
 once
 we
 let
 go
 of
 those
 expectations,
 we
 started
 to
 
find
 some
 brilliant
 situations.
 

 
I
 knew
 that
 time
 would
 be
 an
 issue
 for
 me
 personally
 all
 year
 but
 hoped
 to
 improve.
 In
 
reality
 I
 am
 still
 frequently
 up
 against
 the
 clock
 but
 I
 have
 also
 learnt
 that
 the
 problem
 
only
  comes
  when
  the
  panic
  and
  react
  adversely
  to
  this
  instead
  of
  taking
  the
  opportunity
 
to
  use
  that
  adrenalin
  to
  fuel
  work.
  Thus
  giving
  an
  example
  of
  limitation
  fostering
 
creativity.
 

 
Lack
 of
 experience
 in
 film
 audio
 was
 a
 challenge.
 I
 arrived
 certain
 that
 I
 could
 apply
 my
 
studio
 etiquette
 and
 techniques
 and
 whilst
 true
 to
 some
 extent,
 I
 found
 I
 had
 to
 learn
 a
 
whole
 new
 vocabulary,
 style
 of
 working,
 people
 skills
 to
 succeed.
 I
 particularly
 enjoyed
 
the
 challenge
 of
 working
 with
 people
 we
 met
 and
 documenting
 their
 reality
 rather
 than
 
the
 ‘artist’
 in
 a
 studio
 documenting
 their
 art.
 

 
The
  biggest
  challenge
  has
  been
  self-­‐belief,
  whilst
  learning
  so
  many
  new
  skills,
  it
  was
 
easy
  to
  conclude
  I
  simply
  couldn’t
  do
  anything
  very
  well.
  Accepting
  that
  with
  learning
 
new
 skills
 comes
 the
 responsibility
 of
 understanding
 that
 you
 have
 a
 lot
 more
 to
 learn
 
has
 been
 a
 great
 challenge
 to
 overcome.
 


 

7
 

The
 future
 and
 further
 plans
 for
 the
 work
 

 
I
 hope
 to
 take
 all
 the
 skills
 I
 have
 learned
 this
 year
 and
 continue
 to
 design
 and
 program
 
innovative
  uses
  of
  technology
  to
  both
  enhance
  reality
  and
  encourage
  real
  interactivity
 
and
 immersion
 for
 anyone
 without
 requiring
 technical
 knowledge.
 

 
The
  first
  element
  to
  complete
  after
  graduation
  will
  be
  my
  full
  intention
  for
  the
  3rd
  movement.
 
It
  will
  have
  the
  ability
  to
  become
  heavily
  interactive
  allowing
  the
  guest
  to
  take
  the
  role
  of
  the
 
primary
  orator.
  The
  programming
  of
  this
  movement
  is
  such
  that
  anyone
  can
  stand
  in
  front
  of
 
it,
  offer
  their
  musical
  ideas
  and
  their
  ‘performance
  partner,’
  Max
  MSP,
  will
  respond
  and
 
interact
  in
  a
  sensitive
  way.
  For
  this
  reason
  I
  will
  be
  able
  to
  also
  setup
  this
  movement
  as
  a
 
stand-­‐alone
 installation
 piece
 to
 see
 if
 people
 enjoy
 interacting
 with
 it.
 This
 observation
 will
 
be
 extremely
 useful
 in
 development
 of
 future
 interactive
 works
 also.
 

 
The
 next
 stage
 of
 development
 for
 this
 piece
 –
 and
 an
 exploration
 for
 future
 works
 using
 
similar
  technology
  –
  would
  be
  to
  give
  the
  audience
  more
  control
  over
  the
  dialogue
 
narrative
  and
  let
  them
  choose
  their
  own
  path.
 
  Each
  audience
  member
  would
  be
  given
  a
 
‘passport’
 which
 can
 ‘read’
 the
 objects
 and
 provide
 personal
 narrative
 via
 headphones.
 
The
 passport
 would
 hold
 an
 app
 to
 gather
 audience
 decision-­‐making,
 so
 I
 must
 have
 a
 
way
  to
  send
  that
  information
  from
  the
  interactive
  physical
  element
  to
  the
  app.
  I
  can
 
achieve
 this
 using
 a
 Raspberry,
 a
 RPi-­‐Arduino
 shield
 bridge
 and
 the
 Arduino
 NFC
 shield.
 
 

 
This
 shield
 can
 read
 NFC
 tags
 on
 the
 objects,
 send
 information
 to
 Max
 via
 Maxuino
 and
 
thus
 trigger
 the
 appropriate
 audio
 file.
 An
 extension
 would
 be
 to
 run
 something
 akin
 to
 
wordclock
  over
  a
  wifi
  network
  to
  each
  ‘passport’,
  which
  could
  allow
  more
  specific
 
control
 over
 integration
 of
 the
 headphone
 audio
 with
 the
 speaker
 audio.
 Adding
 a
 WiFI
 
USB
 dongle
 to
 the
 Raspberry
 Pi
 and
 writing
 the
 patch
 to
 have
 an
 overall
 clock
 keeping
 


 

8
 

an
 eye
 on
 the
 piece
 and
 communicating
 musical
 time
 to
 each
 device
 would
 achieve
 this.
 
 
An
 alternative
 to
 the
 Raspberry
 Pi
 would
 be
 to
 use
 an
 Arduino
 Yun
 board
 although
 more
 
research
 is
 needed
 to
 define
 which
 would
 ultimately
 be
 more
 suitable
 for
 the
 task.
 

 
I
  would
  have
  two
  options
  for
  headphones.
  One
  is
  to
  use
  bone-­‐conduction
  technology
  for
 
the
 full
 effect
 of
 both
 audio
 sources
 being
 experienced
 as
 intended
 without
 interference
 
from
 having
 headphones
 over
 ears.
 I
 have
 successfully
 made
 a
 prototype
 of
 simple
 bone-­‐
conducting
 headphones
 made
 from
 piezo
 transducers.
 They
 function
 well
 as
 a
 prototype
 
although
  the
  sound
  quality
  is
  somewhat
  low
  fidelity
  and
  I
  will
  look
  into
  the
  best
 
amplifier
  to
  use,
  as
  the
  one
  I
  used
  in
  the
  prototype
  is
  not
  powerful
  enough.
  I
  could
 
instead
 provide
 users
 with
 an
 alternative
 solution,
 possibly
 being
 open-­‐backed
 earbuds
 
or
  just
  one
  earpiece.
  Whichever
  my
  decision,
  I
  also
  must
  decide
  if
  the
  headphone
  feed
 
will
  be
  mono
  or
  stereo.
  The
  dialogue
  will
  be
  recorded
  using
  my
  own
  binaural
  in-­‐ear
 
microphones
 so
 stereo
 would
 be
 ideal
 but
 perhaps
 overkill
 realistically
 speaking.
 

 
The
  next
  project
  I
  would
  like
  to
  work
  on
  using
  the
  technology
  I
  learn
  through
  this
  study
 
would
 be
 trying
 the
 same
 ideas
 but
 with
 pre-­‐existing
 stories
 and
 fairytales.
 I
 would
 like
 
to
  focus
  on
  bringing
  together
  traditional
  theatre
  elements
  and
  new
  interaction
 
technology
 ideas
 to
 provider
 hyper-­‐immersive
 interactive
 theatre
 that
 the
 audience
 can
 
take
  true
  ownership
  in.
  This
  was
  a
  project
  too
  extensive
  for
  this
  year
  and
  will
  require
 
me
 to
 find
 some
 likeminded
 people
 who
 can
 offer
 their
 specialist
 skills
 to
 help
 out
 in
 the
 
realization.
  Through
  The
  Walk
  to
  Fisterra
  project
  I
  have
  started
  to
  network
 
internationally
 and
 I
 am
 starting
 to
 see
 that
 it
 is
 a
 possibility
 to
 achieve
 this.
 

 

 

 

 


 

9
 

Conclusion
 

 
I
  had
  no
  idea
  the
  joy
  that
  I
  would
  experience
  in
  capturing
  the
  emotions
  of
  others
  in
 
audio.
  Spoken
  word
  as
  music
  has
  always
  been
  a
  interesting
  concept
  to
  me,
  sparked
 
originally
 by
 Steve
 Reich’s
 Different
 Trains
 and
 I
 now
 realise
 that
 it
 is
 not
 just
 a
 simple
 
interest
 but
 I
 really
 love
 to
 listen
 and
 work
 with
 spoken
 word.
 

 
I
 also
 have
 a
 new
 appreciation
 for
 the
 impact
 of
 visual
 media
 and
 I
 will
 be
 making
 efforts
 
to
  further
  my
  familiarity
  with
  this
  medium.
  I
  will
  be
  undertaking
  an
  internship
  in
  the
 
audio
  department
  of
  a
  post-­‐production
  house
  in
  New
  York
  this
  Fall
  which
  I
  anticipate
 
being
 a
 very
 enlightening
 experience
 which
 I
 can
 use
 to
 not
 only
 work
 with
 other
 people
 
but
 also
 implement
 in
 my
 own
 work.
 

 
In
  conclusion,
  my
  primary
  findings
  from
  this
  year
  of
  study
  have
  been
  confirming
  my
 
idea
  that
 technology
 can
 be
 used
 discretely
 and
 simply
 to
 enhance
 one’s
 experience
 of
 
the
  world
  around
  you.
  Now
  I
  hope
  to
  continue
  bringing
  the
  audience
  further
  into
  that
 
world
 with
 fewer
 barriers
 to
 their
 full
 experience.
 

 

 

 

 

 

 

 

 

 

 


 

10