admin
Fri, 10/14/2022 - 16:06
Edited Text
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
CULMINATING EXPERIENCE
FINAL DOCUMENTATION
M.M. (Music Technology Innovation)
Austin Har 763777
1
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
CONTENTS
• Introduction
• Description
• Innovative
• New
of my Culminating Experience Project
Aspects of my Project
Skills Acquired
• Challenges,
• Future
both anticipated and unexpected
Ramifications and/or plans for my project
• Conclusion
• Footnotes
_________________________________________________________________________________________
2
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
1. Introduction
“The enemy of art is the absence of limitations,” Orson Welles.
At Berklee, I found myself surrounded with performers, composers and technologists from around the world.
Each had their unique experiences and perspectives towards music, its cultural value and future trajectories
amidst new and groundbreaking technologies. It became clear to me that I wanted my Culminating Experience
to contribute to this creative zeitgeist by exploring the innovative possibilities of music technology as I became
aware of the importance of capturing and reliving our emotional journeys and inspirations. Thus, as I conversed
with my fellow colleagues, I was inspired by their abundance of creative energy and I wanted to create
something that would be beneficial to our artistic endeavors.
As I learned new software for my coursework, I began to ponder how the songwriting process had evolved over
time with new technologies. What tools did songwriters use to write music today? How did they transfer ideas
and files from one place to another? How do we capture that moment of inspiration that would eventually lead
to the creation of a work of art? These were the questions that I found myself seeking the answers to as I
conceptualized my project. Throughout my investigation, I found room for innovation and meaningful
contribution to the field of music technology in the project that I decided to commit my efforts to.
_________________________________________________________________________________________
2. Description of Culminating Experience
“For creativity on the go, capture your inspiration with FLO.”
‘FLO’ is an iOS app for the everyday songwriter who needs a simple interface for organizing their daily
creativity and capturing that moment of inspiration. Professional and amateur songwriters, composers and
music creators will be able to use ‘FLO’ on their iPhone or iPad to record, store and transfer their everyday
ideas online as MIDI files. Whether you are taking a stroll in the park or sitting at home watching TV,
songwriters can capture their moment of inspiration and transfer it online for further development.
3
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Working logo for ‘FLO’)
‘FLO’ contains 3 essential tools for capturing your ideas:
1. Drum machine
2. Synthesizer
3. Recorder (via the iPhone / iPad microphone)
These three features were designed with the utmost consideration for the modern songwriter’s needs and
intuitions during their creative process. The various capabilities of each feature are listed below:
1. Drum Machine:
- 32 step drum sequencer for kick, snare, hi hat and other samples.
- Beat quantization patch to the nearest 8th note value for real time recordings.
- MIDI export and online upload onto Parse server for further development on
other DAWs such as Ableton and Pro Tools.
2. Synthesizer:
- Integration into Pure Data patch for real time synthesis.
- Beat quantization patch to the nearest 8th note value for real time recordings.
4
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
- MIDI export and online upload onto Parse server for further development on
other DAWs such as Ableton and Pro Tools.
3. Recorder:
- Integration into Pure Data patch for Audio to MIDI conversion.
-
MIDI export and online upload onto Parse server for further development on
other DAWs such as Ableton and Pro Tools.
_________________________________________________________________________________________
UI Design
The UI design of ‘FLO’ began from early prototypes in MAX MSP. As I prototyped the final designs with the
standard Xcode storyboard, I discovered a very useful OSX application called ‘PaintCode’.
(My drum icon design in ‘PaintCode’)
‘PaintCode’ is a powerful application that allows users to create intricate icons, buttons or interfaces, or
alternatively to import svg and Photoshop files, that are automatically converted into usable Objective C code.
5
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
By drawing the icons, buttons and UI as Objective C code, rather than using conventional .png files, this saves
a significant amount of storage space in the app.
_________________________________________________________________________________________
Programming Languages Used
For the significant programming portion of the app, I used three primary languages:
1. MAX MSP:
- Object oriented music technology programming language.
- Original working prototypes of calendar and diary storage functions.
(Early diary interface prototype)
6
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Early diary interface prototype)
2. Pure Data:
- Object oriented music technology programming language with iOS function.
- Drum machine, BPM, Audio to MIDI converter, Beat Quantization,
Synthesizer, MIDI file making and Xcode Control patches.
(Drum machine patch)
My Drum Machine patch features a 32-step drum sequencer, which receives a message of ‘1’ or ‘0’ for each
designated kick, snare or hi hat selection. The patch receives the BPM float designated by the user. Listening to
the click track, the user records their drum sequence and is able to save, playback and export it as a MIDI file.
7
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(BPM patch)
My BPM patch converts the designated BPM into milliseconds for the real time beat quantization with the click
track and 32-step drum machine grid. This patch and its corresponding click track are crucial to the user’s
recording process, whether that be with the drum machine, synthesizer or recorder.
(Audio to MIDI converter patch)
My Audio to MIDI converter patch is perhaps the most ambitious one I programmed in Pure Data. It utilizes the
‘fiddle’ object to analyze incoming audio data into frequencies and MIDI pitch values. Using an amplitude
filter (to cancel background ambience) and the ‘timer’ object, I was able to deduce the duration of each
incoming note. The patch converts an audio recording of a guitar melody of up to 32 distinct notes, where the
8
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
user is then able to save, playback and export it as a MIDI file.
(Beat Quantization patch)
My Beat Quantization patch receives the millisecond duration of each ‘beat’ from the BPM patch, which allows
it to quantize real time inputs to the nearest 8th note. Using the ‘timer’ and ‘&&’ objects, the logic behind my
beat quantization patch lies in dividing each beat duration in half and then filtering those inputs which are less
than half a beat early or late to the nearest beat value. The quantization is visible in the resulting Drum Machine
patch where the quantized beats are marked as selected toggles.
(Synthesizer patch)
9
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
My Synthesizer patch allows the user to select a sine, square or triangle wave (or various combinations of the
three). The original prototype utilized the ‘keyin’ object, which tracked the user’s ‘keydown’ events from the
letters A to K (representing an octave from C). The final synthesizer patch receives messages from the Xcode
storyboard corresponding to a frequency to MIDI key, which in turn triggers the corresponding pitch selected
from the original system. The user is then able to save, playback and export it as a MIDI file.
(MIDI file making patch)
The ‘midifile’ object is a crucial part of my app’s functionality. This is the patch that collects all the MIDI pitch
and duration data from the drum machine, synthesizer and recorder, and then compiles it into a .mid file.
‘Midifile’ is a Pure Data external which required a very extensive implementation and compilation process. I
modified this patch to automatically update the user id of each MIDI recording.
10
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Xcode Control patch)
My Xcode Control patch is the main patch for receiving and sending data between Pure Data and Xcode via the
libpd library. Floats, messages and symbols are transferred here to other Pure Data patches such as the Drum
Machine and Synthesizer, whilst the ‘.mid’ file compiled in the ‘midifile’ patch is sent to Xcode to be saved in
the app’s document’s directory folder and also uploaded online via the Parse server API.
3. Objective C:
- C based programming language in Xcode used for iOS development
-
Drum Machine, Synthesizer, Recorder, PaintCode UI, Document’s Directory,
libpd library, .plist resource and Parse API.
(App Delegate .m file)
11
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Document’s Directory initialization code)
The Parse API and Document’s Directory initialization code is called with the first method ‘drawRect’, along
with the UI code from PaintCode. The NSBundle object allocates memory space for the MIDI data created in
‘FLO’ via the Pure Data ‘midifile’ object that is connected to Xcode through the libpd library. This data is then
stored into the app’s document’s directory folder and uploaded into the Parse server via the PFObject function.
(Parse API Framework)
The Parse framework is imported into the relevant ‘h’ and ‘m’ files of the various views so that the MIDI data
created in each of the functions (drum machine, synthesizer and recorder) will be able to be uploaded directly
onto the server (as seen below).
12
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Parse server database log)
(iOS Storyboard)
As I used ‘PaintCode’ for a code generated UI on each of these view controllers, I placed a ‘View’ object on
each of the view controllers to receive and send events from the iOS screen. Essentially, the ‘View’ object acts
as a screen for the ‘PaintCode’ UI to be displayed.
13
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
3. Innovative Aspects of the Work
‘FLO’ is innovative and differs in comparison to successful apps such as 'Day One', 'Live Journal' and
‘Evernote’ (note taking apps), Native Instruments ‘iMaschine’ and Propellerhead ‘Figure’ (beat making apps)
in three fundamental aspects:
1. Template for music creation, storage and transfer: ‘FLO’ offers a music specific template for not
only creating, but also storing and transferring ideas for further development on common DAWs such as
Ableton or Pro Tools. Current beat making apps function by having the entire track produced on the app
itself, then exporting it as a single bounced audio file. This only allows very limited options for further
development on professional DAWs, whereas the MIDI capability of ‘FLO’ is a significant advantage
for further creative development.
2. Audio to MIDI conversion: ‘FLO’ features a highly appealing Audio to MIDI conversion feature,
which is not currently found in major music apps (and those which attempted it have received largely
negative reviews). ‘MIDIMorphosis’ is by far the best-received iOS MIDI conversion app but is made
specifically for guitar and bass only.
14
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(MIDImorphosis on the App Store)
A raw guitar sound is a much purer tone than a human voice, thus there are significantly fewer pitch
frequencies for the app to analyze. Furthermore, there are technological considerations regarding monophonic,
homophonic and polyphonic analysis; the latter of which is also significantly easier with a pure tone such as
guitar or bass. Whilst software such as Celemony’s Melodyne and Neptune’s Autotune are able to do Audio to
MIDI conversion well, it is difficult to replicate their success on an iOS app due to the complexity of this
technology and the audio quality of the iPhone and iPad microphone for recording raw sounds.
(Celemony Melodyne’s pitch correction feature)
15
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Neptune Autotune’s pitch correction feature)
Audio to MIDI conversion involves complex amplitude and spectral analysis of frequency data from an audio
source in order to determine a note’s pitch, length and velocity. These processes require not only audio
processing and mathematical operations, but also file management, storage and transfer, which require the
combination of several programming languages. With ‘FLO’, I was required to implement Pure Data, Objective
C and C++, along with their corresponding object libraries and foundations, libpd for Pure Data, Parse
Foundation and AV Foundation for Xcode.
As of today, ‘FLO’ is able to replicate the efficiency of Audio to MIDI conversion with as much success as
‘MIDIMorphosis’ on guitar for its monophonic capabilities. As ‘FLO’ is predominantly designed for quick
‘MIDI note taking’, homophonic and polyphonic capabilities were not my top priority.
3. MIDI note taking: Unlike ‘iMaschine’ and ‘Figure’ which only transfer audio files via USB, ‘FLO’
can transfer MIDI files of beats and melodies as well audio recordings and photos online via the Parse
API. MIDI files are superior to audio files in terms of size, transfer speed and flexibility for
manipulation.
(Native Instrument iMaschine’s UI design)
16
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Propellerhead Figure’s UI design)
Based on my research with Berklee business majors Carl and Tanya into competing apps, we found few
songwriting apps that compete with ‘FLO’s ‘MIDI note taking’ and Audio to MIDI conversion features.
(Survey for ‘FLO’)
The survey that I conducted with my business partners confirmed the demand for these innovative features that
‘FLO’ offered. As seen in the survey responses, creating and sharing MIDI files proved to be a major aspect of
compositional and collaboration process for modern day songwriters.
17
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Survey for ‘FLO’)
_________________________________________________________________________________________
4. New Skills Acquired
I acquired many new skills for my Culminating Experience project. I had no programming experience prior to
my studies at Berklee and I was required to learn several languages in order to program my iOS app:
1. MAX MSP
2. Pure Data
3. Objective C
It is also worth mentioning the different capacities in which I used these programming languages. Not only did
I grow familiar with these programming languages, I also developed the fundamental skills and problem
solving logic for conceptualizing, designing and executing an original iOS application from scratch:
1. Implementation of Parse API into Objective C
2. Beat Quantization logic
3. Audio to MIDI conversion logic
4. Real time synthesis
5. Implementation of external objects and libraries (‘midifile’ object and ‘libpd’ library)
6. Communication between several programming languages, object libraries, frameworks and servers
18
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
Asides from the new skills I gained in programming, I also gained new skills with software such as Ableton
(which I had no prior experience), that provided the initial inspiration for ‘FLO’s MIDI note-taking function.
Other skills included:
1. Independent research through online forums and tutorials
2. Time management skills
3. Presentation skills
4. Collaboration skills
_________________________________________________________________________________________
5. Challenges Encountered
As my Culminating Experience project involved the extensive use of programming languages, which I had no
prior experience with before coming to Berklee, I faced many challenges throughout the entire process. During
the initial conceptualization stages, the biggest challenge I faced was the need to be innovative. This challenge
was especially difficult given the many new and innovative technologies emerging for iOS applications today.
With guidance from my program director Stephen Webber, advisor Ben Hogue and visiting professors David
Mash and Richard Boulanger, I was able to incorporate the defining characteristics of an innovative product
into my final app.
Upon reaching a focused objective for my project, the process was then complicated by our substantial
coursework and my limited programming skills. Throughout the entire execution process, the steep learning
curve required for improving one’s programming skills became evident and I revised my earlier ambitious
vision for more specific weekly goals given the short timeline. As I developed my skills, I was then able to
develop a more realistic vision for my final application.
_________________________________________________________________________________________
6. Future Ramifications for the Project
In our modern digital information age, recording personal experiences and sharing them with the world via
social networking is increasingly expected of us. ‘FLO’ has the potential to contribute to this 21st century
19
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
zeitgeist by allowing creative content to be recorded, stored and transferred as MIDI files online. This would
open new and exciting doors for creative collaborations between ‘FLO’ users as their ideas could potentially be
integrated as a feature on social networking platforms such as Facebook or Twitter via integrating the relevant
API into its code.
As indicated in my survey, the majority of musicians collaborate with each other via sharing files such as MIDI,
audio samples and text. This shows the potential for ‘FLO’ to blossom into a larger scale project with a team of
programmers and established business partners (i.e. Ableton, Pro Tools, Facebook) to develop a useful and
innovative format for everyday users to share their musical notes online. Furthermore, visiting artists such as DJ
Guru stated that a MIDI note-taking iOS application could become the ‘standard’ for capturing musical ideas.
_________________________________________________________________________________________
7. Conclusion
In summary, ‘FLO’ is designed to provide the utmost speed and efficiency in capturing the basic essential
details of a songwriter’s daily inspirations. Through its 3 features, ‘FLO’ allows users to quickly record, store
and transfer musical ideas as MIDI files onto DAWs, such as Ableton and Pro Tools, for further development
through the Parse server API. From here, users can import that MIDI data onto their computer to start or
continue working on a bigger project such as a musical composition or track.
Creating ‘FLO’ was a very challenging and fulfilling experience for me. Not only did I acquire a varied range
of new skills, I was also able to open my mind to different perspectives of music technology as a developer and
user. This was a new and ambitious project for me and I am thankful for all the inspiration and guidance that I
received during my studies at Berklee.
20
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
7. Footnotes
The following are the main sources from which I developed my programming skills, along with consultations
from my supervisor Ben Hogue:
Peter Brinkmann, 2012. Real-Time Audio Synthesis on Android and iOS: Making Musical Apps. O-Reilly
Media Inc.
www.stackoverflow.com (online question and answer forum for programmers)
www.lynda.com
www.youtube.com
http://ipgames.wordpress.com/tutorials/writeread-data-to-plist-file/
http://www.raywenderlich.com/36341/paintcode-tutorial-dynamic-buttons
https://developer.apple.com/
https://www.parse.com/
http://en.flossmanuals.net/csound/e-csound-in-ios/
https://github.com/libpd/pd-for-ios/wiki/ios
_________________________________________________________________________________________
_________________________________________________________________________________________
21
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
CULMINATING EXPERIENCE
FINAL DOCUMENTATION
M.M. (Music Technology Innovation)
Austin Har 763777
1
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
CONTENTS
• Introduction
• Description
• Innovative
• New
of my Culminating Experience Project
Aspects of my Project
Skills Acquired
• Challenges,
• Future
both anticipated and unexpected
Ramifications and/or plans for my project
• Conclusion
• Footnotes
_________________________________________________________________________________________
2
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
1. Introduction
“The enemy of art is the absence of limitations,” Orson Welles.
At Berklee, I found myself surrounded with performers, composers and technologists from around the world.
Each had their unique experiences and perspectives towards music, its cultural value and future trajectories
amidst new and groundbreaking technologies. It became clear to me that I wanted my Culminating Experience
to contribute to this creative zeitgeist by exploring the innovative possibilities of music technology as I became
aware of the importance of capturing and reliving our emotional journeys and inspirations. Thus, as I conversed
with my fellow colleagues, I was inspired by their abundance of creative energy and I wanted to create
something that would be beneficial to our artistic endeavors.
As I learned new software for my coursework, I began to ponder how the songwriting process had evolved over
time with new technologies. What tools did songwriters use to write music today? How did they transfer ideas
and files from one place to another? How do we capture that moment of inspiration that would eventually lead
to the creation of a work of art? These were the questions that I found myself seeking the answers to as I
conceptualized my project. Throughout my investigation, I found room for innovation and meaningful
contribution to the field of music technology in the project that I decided to commit my efforts to.
_________________________________________________________________________________________
2. Description of Culminating Experience
“For creativity on the go, capture your inspiration with FLO.”
‘FLO’ is an iOS app for the everyday songwriter who needs a simple interface for organizing their daily
creativity and capturing that moment of inspiration. Professional and amateur songwriters, composers and
music creators will be able to use ‘FLO’ on their iPhone or iPad to record, store and transfer their everyday
ideas online as MIDI files. Whether you are taking a stroll in the park or sitting at home watching TV,
songwriters can capture their moment of inspiration and transfer it online for further development.
3
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Working logo for ‘FLO’)
‘FLO’ contains 3 essential tools for capturing your ideas:
1. Drum machine
2. Synthesizer
3. Recorder (via the iPhone / iPad microphone)
These three features were designed with the utmost consideration for the modern songwriter’s needs and
intuitions during their creative process. The various capabilities of each feature are listed below:
1. Drum Machine:
- 32 step drum sequencer for kick, snare, hi hat and other samples.
- Beat quantization patch to the nearest 8th note value for real time recordings.
- MIDI export and online upload onto Parse server for further development on
other DAWs such as Ableton and Pro Tools.
2. Synthesizer:
- Integration into Pure Data patch for real time synthesis.
- Beat quantization patch to the nearest 8th note value for real time recordings.
4
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
- MIDI export and online upload onto Parse server for further development on
other DAWs such as Ableton and Pro Tools.
3. Recorder:
- Integration into Pure Data patch for Audio to MIDI conversion.
-
MIDI export and online upload onto Parse server for further development on
other DAWs such as Ableton and Pro Tools.
_________________________________________________________________________________________
UI Design
The UI design of ‘FLO’ began from early prototypes in MAX MSP. As I prototyped the final designs with the
standard Xcode storyboard, I discovered a very useful OSX application called ‘PaintCode’.
(My drum icon design in ‘PaintCode’)
‘PaintCode’ is a powerful application that allows users to create intricate icons, buttons or interfaces, or
alternatively to import svg and Photoshop files, that are automatically converted into usable Objective C code.
5
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
By drawing the icons, buttons and UI as Objective C code, rather than using conventional .png files, this saves
a significant amount of storage space in the app.
_________________________________________________________________________________________
Programming Languages Used
For the significant programming portion of the app, I used three primary languages:
1. MAX MSP:
- Object oriented music technology programming language.
- Original working prototypes of calendar and diary storage functions.
(Early diary interface prototype)
6
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Early diary interface prototype)
2. Pure Data:
- Object oriented music technology programming language with iOS function.
- Drum machine, BPM, Audio to MIDI converter, Beat Quantization,
Synthesizer, MIDI file making and Xcode Control patches.
(Drum machine patch)
My Drum Machine patch features a 32-step drum sequencer, which receives a message of ‘1’ or ‘0’ for each
designated kick, snare or hi hat selection. The patch receives the BPM float designated by the user. Listening to
the click track, the user records their drum sequence and is able to save, playback and export it as a MIDI file.
7
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(BPM patch)
My BPM patch converts the designated BPM into milliseconds for the real time beat quantization with the click
track and 32-step drum machine grid. This patch and its corresponding click track are crucial to the user’s
recording process, whether that be with the drum machine, synthesizer or recorder.
(Audio to MIDI converter patch)
My Audio to MIDI converter patch is perhaps the most ambitious one I programmed in Pure Data. It utilizes the
‘fiddle’ object to analyze incoming audio data into frequencies and MIDI pitch values. Using an amplitude
filter (to cancel background ambience) and the ‘timer’ object, I was able to deduce the duration of each
incoming note. The patch converts an audio recording of a guitar melody of up to 32 distinct notes, where the
8
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
user is then able to save, playback and export it as a MIDI file.
(Beat Quantization patch)
My Beat Quantization patch receives the millisecond duration of each ‘beat’ from the BPM patch, which allows
it to quantize real time inputs to the nearest 8th note. Using the ‘timer’ and ‘&&’ objects, the logic behind my
beat quantization patch lies in dividing each beat duration in half and then filtering those inputs which are less
than half a beat early or late to the nearest beat value. The quantization is visible in the resulting Drum Machine
patch where the quantized beats are marked as selected toggles.
(Synthesizer patch)
9
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
My Synthesizer patch allows the user to select a sine, square or triangle wave (or various combinations of the
three). The original prototype utilized the ‘keyin’ object, which tracked the user’s ‘keydown’ events from the
letters A to K (representing an octave from C). The final synthesizer patch receives messages from the Xcode
storyboard corresponding to a frequency to MIDI key, which in turn triggers the corresponding pitch selected
from the original system. The user is then able to save, playback and export it as a MIDI file.
(MIDI file making patch)
The ‘midifile’ object is a crucial part of my app’s functionality. This is the patch that collects all the MIDI pitch
and duration data from the drum machine, synthesizer and recorder, and then compiles it into a .mid file.
‘Midifile’ is a Pure Data external which required a very extensive implementation and compilation process. I
modified this patch to automatically update the user id of each MIDI recording.
10
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Xcode Control patch)
My Xcode Control patch is the main patch for receiving and sending data between Pure Data and Xcode via the
libpd library. Floats, messages and symbols are transferred here to other Pure Data patches such as the Drum
Machine and Synthesizer, whilst the ‘.mid’ file compiled in the ‘midifile’ patch is sent to Xcode to be saved in
the app’s document’s directory folder and also uploaded online via the Parse server API.
3. Objective C:
- C based programming language in Xcode used for iOS development
-
Drum Machine, Synthesizer, Recorder, PaintCode UI, Document’s Directory,
libpd library, .plist resource and Parse API.
(App Delegate .m file)
11
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Document’s Directory initialization code)
The Parse API and Document’s Directory initialization code is called with the first method ‘drawRect’, along
with the UI code from PaintCode. The NSBundle object allocates memory space for the MIDI data created in
‘FLO’ via the Pure Data ‘midifile’ object that is connected to Xcode through the libpd library. This data is then
stored into the app’s document’s directory folder and uploaded into the Parse server via the PFObject function.
(Parse API Framework)
The Parse framework is imported into the relevant ‘h’ and ‘m’ files of the various views so that the MIDI data
created in each of the functions (drum machine, synthesizer and recorder) will be able to be uploaded directly
onto the server (as seen below).
12
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Parse server database log)
(iOS Storyboard)
As I used ‘PaintCode’ for a code generated UI on each of these view controllers, I placed a ‘View’ object on
each of the view controllers to receive and send events from the iOS screen. Essentially, the ‘View’ object acts
as a screen for the ‘PaintCode’ UI to be displayed.
13
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
3. Innovative Aspects of the Work
‘FLO’ is innovative and differs in comparison to successful apps such as 'Day One', 'Live Journal' and
‘Evernote’ (note taking apps), Native Instruments ‘iMaschine’ and Propellerhead ‘Figure’ (beat making apps)
in three fundamental aspects:
1. Template for music creation, storage and transfer: ‘FLO’ offers a music specific template for not
only creating, but also storing and transferring ideas for further development on common DAWs such as
Ableton or Pro Tools. Current beat making apps function by having the entire track produced on the app
itself, then exporting it as a single bounced audio file. This only allows very limited options for further
development on professional DAWs, whereas the MIDI capability of ‘FLO’ is a significant advantage
for further creative development.
2. Audio to MIDI conversion: ‘FLO’ features a highly appealing Audio to MIDI conversion feature,
which is not currently found in major music apps (and those which attempted it have received largely
negative reviews). ‘MIDIMorphosis’ is by far the best-received iOS MIDI conversion app but is made
specifically for guitar and bass only.
14
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(MIDImorphosis on the App Store)
A raw guitar sound is a much purer tone than a human voice, thus there are significantly fewer pitch
frequencies for the app to analyze. Furthermore, there are technological considerations regarding monophonic,
homophonic and polyphonic analysis; the latter of which is also significantly easier with a pure tone such as
guitar or bass. Whilst software such as Celemony’s Melodyne and Neptune’s Autotune are able to do Audio to
MIDI conversion well, it is difficult to replicate their success on an iOS app due to the complexity of this
technology and the audio quality of the iPhone and iPad microphone for recording raw sounds.
(Celemony Melodyne’s pitch correction feature)
15
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Neptune Autotune’s pitch correction feature)
Audio to MIDI conversion involves complex amplitude and spectral analysis of frequency data from an audio
source in order to determine a note’s pitch, length and velocity. These processes require not only audio
processing and mathematical operations, but also file management, storage and transfer, which require the
combination of several programming languages. With ‘FLO’, I was required to implement Pure Data, Objective
C and C++, along with their corresponding object libraries and foundations, libpd for Pure Data, Parse
Foundation and AV Foundation for Xcode.
As of today, ‘FLO’ is able to replicate the efficiency of Audio to MIDI conversion with as much success as
‘MIDIMorphosis’ on guitar for its monophonic capabilities. As ‘FLO’ is predominantly designed for quick
‘MIDI note taking’, homophonic and polyphonic capabilities were not my top priority.
3. MIDI note taking: Unlike ‘iMaschine’ and ‘Figure’ which only transfer audio files via USB, ‘FLO’
can transfer MIDI files of beats and melodies as well audio recordings and photos online via the Parse
API. MIDI files are superior to audio files in terms of size, transfer speed and flexibility for
manipulation.
(Native Instrument iMaschine’s UI design)
16
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Propellerhead Figure’s UI design)
Based on my research with Berklee business majors Carl and Tanya into competing apps, we found few
songwriting apps that compete with ‘FLO’s ‘MIDI note taking’ and Audio to MIDI conversion features.
(Survey for ‘FLO’)
The survey that I conducted with my business partners confirmed the demand for these innovative features that
‘FLO’ offered. As seen in the survey responses, creating and sharing MIDI files proved to be a major aspect of
compositional and collaboration process for modern day songwriters.
17
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
(Survey for ‘FLO’)
_________________________________________________________________________________________
4. New Skills Acquired
I acquired many new skills for my Culminating Experience project. I had no programming experience prior to
my studies at Berklee and I was required to learn several languages in order to program my iOS app:
1. MAX MSP
2. Pure Data
3. Objective C
It is also worth mentioning the different capacities in which I used these programming languages. Not only did
I grow familiar with these programming languages, I also developed the fundamental skills and problem
solving logic for conceptualizing, designing and executing an original iOS application from scratch:
1. Implementation of Parse API into Objective C
2. Beat Quantization logic
3. Audio to MIDI conversion logic
4. Real time synthesis
5. Implementation of external objects and libraries (‘midifile’ object and ‘libpd’ library)
6. Communication between several programming languages, object libraries, frameworks and servers
18
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
Asides from the new skills I gained in programming, I also gained new skills with software such as Ableton
(which I had no prior experience), that provided the initial inspiration for ‘FLO’s MIDI note-taking function.
Other skills included:
1. Independent research through online forums and tutorials
2. Time management skills
3. Presentation skills
4. Collaboration skills
_________________________________________________________________________________________
5. Challenges Encountered
As my Culminating Experience project involved the extensive use of programming languages, which I had no
prior experience with before coming to Berklee, I faced many challenges throughout the entire process. During
the initial conceptualization stages, the biggest challenge I faced was the need to be innovative. This challenge
was especially difficult given the many new and innovative technologies emerging for iOS applications today.
With guidance from my program director Stephen Webber, advisor Ben Hogue and visiting professors David
Mash and Richard Boulanger, I was able to incorporate the defining characteristics of an innovative product
into my final app.
Upon reaching a focused objective for my project, the process was then complicated by our substantial
coursework and my limited programming skills. Throughout the entire execution process, the steep learning
curve required for improving one’s programming skills became evident and I revised my earlier ambitious
vision for more specific weekly goals given the short timeline. As I developed my skills, I was then able to
develop a more realistic vision for my final application.
_________________________________________________________________________________________
6. Future Ramifications for the Project
In our modern digital information age, recording personal experiences and sharing them with the world via
social networking is increasingly expected of us. ‘FLO’ has the potential to contribute to this 21st century
19
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
zeitgeist by allowing creative content to be recorded, stored and transferred as MIDI files online. This would
open new and exciting doors for creative collaborations between ‘FLO’ users as their ideas could potentially be
integrated as a feature on social networking platforms such as Facebook or Twitter via integrating the relevant
API into its code.
As indicated in my survey, the majority of musicians collaborate with each other via sharing files such as MIDI,
audio samples and text. This shows the potential for ‘FLO’ to blossom into a larger scale project with a team of
programmers and established business partners (i.e. Ableton, Pro Tools, Facebook) to develop a useful and
innovative format for everyday users to share their musical notes online. Furthermore, visiting artists such as DJ
Guru stated that a MIDI note-taking iOS application could become the ‘standard’ for capturing musical ideas.
_________________________________________________________________________________________
7. Conclusion
In summary, ‘FLO’ is designed to provide the utmost speed and efficiency in capturing the basic essential
details of a songwriter’s daily inspirations. Through its 3 features, ‘FLO’ allows users to quickly record, store
and transfer musical ideas as MIDI files onto DAWs, such as Ableton and Pro Tools, for further development
through the Parse server API. From here, users can import that MIDI data onto their computer to start or
continue working on a bigger project such as a musical composition or track.
Creating ‘FLO’ was a very challenging and fulfilling experience for me. Not only did I acquire a varied range
of new skills, I was also able to open my mind to different perspectives of music technology as a developer and
user. This was a new and ambitious project for me and I am thankful for all the inspiration and guidance that I
received during my studies at Berklee.
20
MM
(MTI)
Berklee
College
of
Music:
Final
Documentation
Austin
Har
763777
_________________________________________________________________________________________
7. Footnotes
The following are the main sources from which I developed my programming skills, along with consultations
from my supervisor Ben Hogue:
Peter Brinkmann, 2012. Real-Time Audio Synthesis on Android and iOS: Making Musical Apps. O-Reilly
Media Inc.
www.stackoverflow.com (online question and answer forum for programmers)
www.lynda.com
www.youtube.com
http://ipgames.wordpress.com/tutorials/writeread-data-to-plist-file/
http://www.raywenderlich.com/36341/paintcode-tutorial-dynamic-buttons
https://developer.apple.com/
https://www.parse.com/
http://en.flossmanuals.net/csound/e-csound-in-ios/
https://github.com/libpd/pd-for-ios/wiki/ios
_________________________________________________________________________________________
_________________________________________________________________________________________
21
Media of
