MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   CULMINATING EXPERIENCE FINAL DOCUMENTATION M.M. (Music Technology Innovation) Austin Har 763777   1 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   _________________________________________________________________________________________ CONTENTS • Introduction • Description • Innovative • New of my Culminating Experience Project Aspects of my Project Skills Acquired • Challenges, • Future both anticipated and unexpected Ramifications and/or plans for my project • Conclusion • Footnotes _________________________________________________________________________________________   2 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   _________________________________________________________________________________________ 1. Introduction “The enemy of art is the absence of limitations,” Orson Welles. At Berklee, I found myself surrounded with performers, composers and technologists from around the world. Each had their unique experiences and perspectives towards music, its cultural value and future trajectories amidst new and groundbreaking technologies. It became clear to me that I wanted my Culminating Experience to contribute to this creative zeitgeist by exploring the innovative possibilities of music technology as I became aware of the importance of capturing and reliving our emotional journeys and inspirations. Thus, as I conversed with my fellow colleagues, I was inspired by their abundance of creative energy and I wanted to create something that would be beneficial to our artistic endeavors. As I learned new software for my coursework, I began to ponder how the songwriting process had evolved over time with new technologies. What tools did songwriters use to write music today? How did they transfer ideas and files from one place to another? How do we capture that moment of inspiration that would eventually lead to the creation of a work of art? These were the questions that I found myself seeking the answers to as I conceptualized my project. Throughout my investigation, I found room for innovation and meaningful contribution to the field of music technology in the project that I decided to commit my efforts to. _________________________________________________________________________________________ 2. Description of Culminating Experience “For creativity on the go, capture your inspiration with FLO.” ‘FLO’ is an iOS app for the everyday songwriter who needs a simple interface for organizing their daily creativity and capturing that moment of inspiration. Professional and amateur songwriters, composers and music creators will be able to use ‘FLO’ on their iPhone or iPad to record, store and transfer their everyday ideas online as MIDI files. Whether you are taking a stroll in the park or sitting at home watching TV, songwriters can capture their moment of inspiration and transfer it online for further development.   3 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Working logo for ‘FLO’) ‘FLO’ contains 3 essential tools for capturing your ideas: 1. Drum machine 2. Synthesizer 3. Recorder (via the iPhone / iPad microphone) These three features were designed with the utmost consideration for the modern songwriter’s needs and intuitions during their creative process. The various capabilities of each feature are listed below: 1. Drum Machine: - 32 step drum sequencer for kick, snare, hi hat and other samples. - Beat quantization patch to the nearest 8th note value for real time recordings. - MIDI export and online upload onto Parse server for further development on other DAWs such as Ableton and Pro Tools. 2. Synthesizer: - Integration into Pure Data patch for real time synthesis. - Beat quantization patch to the nearest 8th note value for real time recordings.   4 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   - MIDI export and online upload onto Parse server for further development on other DAWs such as Ableton and Pro Tools. 3. Recorder: - Integration into Pure Data patch for Audio to MIDI conversion. - MIDI export and online upload onto Parse server for further development on other DAWs such as Ableton and Pro Tools. _________________________________________________________________________________________ UI Design The UI design of ‘FLO’ began from early prototypes in MAX MSP. As I prototyped the final designs with the standard Xcode storyboard, I discovered a very useful OSX application called ‘PaintCode’. (My drum icon design in ‘PaintCode’) ‘PaintCode’ is a powerful application that allows users to create intricate icons, buttons or interfaces, or alternatively to import svg and Photoshop files, that are automatically converted into usable Objective C code.   5 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   By drawing the icons, buttons and UI as Objective C code, rather than using conventional .png files, this saves a significant amount of storage space in the app. _________________________________________________________________________________________ Programming Languages Used For the significant programming portion of the app, I used three primary languages: 1. MAX MSP: - Object oriented music technology programming language. - Original working prototypes of calendar and diary storage functions. (Early diary interface prototype)   6 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Early diary interface prototype) 2. Pure Data: - Object oriented music technology programming language with iOS function. - Drum machine, BPM, Audio to MIDI converter, Beat Quantization, Synthesizer, MIDI file making and Xcode Control patches. (Drum machine patch) My Drum Machine patch features a 32-step drum sequencer, which receives a message of ‘1’ or ‘0’ for each designated kick, snare or hi hat selection. The patch receives the BPM float designated by the user. Listening to the click track, the user records their drum sequence and is able to save, playback and export it as a MIDI file.   7 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (BPM patch) My BPM patch converts the designated BPM into milliseconds for the real time beat quantization with the click track and 32-step drum machine grid. This patch and its corresponding click track are crucial to the user’s recording process, whether that be with the drum machine, synthesizer or recorder. (Audio to MIDI converter patch) My Audio to MIDI converter patch is perhaps the most ambitious one I programmed in Pure Data. It utilizes the ‘fiddle’ object to analyze incoming audio data into frequencies and MIDI pitch values. Using an amplitude filter (to cancel background ambience) and the ‘timer’ object, I was able to deduce the duration of each incoming note. The patch converts an audio recording of a guitar melody of up to 32 distinct notes, where the   8 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   user is then able to save, playback and export it as a MIDI file. (Beat Quantization patch) My Beat Quantization patch receives the millisecond duration of each ‘beat’ from the BPM patch, which allows it to quantize real time inputs to the nearest 8th note. Using the ‘timer’ and ‘&&’ objects, the logic behind my beat quantization patch lies in dividing each beat duration in half and then filtering those inputs which are less than half a beat early or late to the nearest beat value. The quantization is visible in the resulting Drum Machine patch where the quantized beats are marked as selected toggles. (Synthesizer patch)   9 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   My Synthesizer patch allows the user to select a sine, square or triangle wave (or various combinations of the three). The original prototype utilized the ‘keyin’ object, which tracked the user’s ‘keydown’ events from the letters A to K (representing an octave from C). The final synthesizer patch receives messages from the Xcode storyboard corresponding to a frequency to MIDI key, which in turn triggers the corresponding pitch selected from the original system. The user is then able to save, playback and export it as a MIDI file. (MIDI file making patch) The ‘midifile’ object is a crucial part of my app’s functionality. This is the patch that collects all the MIDI pitch and duration data from the drum machine, synthesizer and recorder, and then compiles it into a .mid file. ‘Midifile’ is a Pure Data external which required a very extensive implementation and compilation process. I modified this patch to automatically update the user id of each MIDI recording.   10 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Xcode Control patch) My Xcode Control patch is the main patch for receiving and sending data between Pure Data and Xcode via the libpd library. Floats, messages and symbols are transferred here to other Pure Data patches such as the Drum Machine and Synthesizer, whilst the ‘.mid’ file compiled in the ‘midifile’ patch is sent to Xcode to be saved in the app’s document’s directory folder and also uploaded online via the Parse server API. 3. Objective C: - C based programming language in Xcode used for iOS development - Drum Machine, Synthesizer, Recorder, PaintCode UI, Document’s Directory, libpd library, .plist resource and Parse API. (App Delegate .m file)   11 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Document’s Directory initialization code) The Parse API and Document’s Directory initialization code is called with the first method ‘drawRect’, along with the UI code from PaintCode. The NSBundle object allocates memory space for the MIDI data created in ‘FLO’ via the Pure Data ‘midifile’ object that is connected to Xcode through the libpd library. This data is then stored into the app’s document’s directory folder and uploaded into the Parse server via the PFObject function. (Parse API Framework) The Parse framework is imported into the relevant ‘h’ and ‘m’ files of the various views so that the MIDI data created in each of the functions (drum machine, synthesizer and recorder) will be able to be uploaded directly onto the server (as seen below).   12 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Parse server database log) (iOS Storyboard) As I used ‘PaintCode’ for a code generated UI on each of these view controllers, I placed a ‘View’ object on each of the view controllers to receive and send events from the iOS screen. Essentially, the ‘View’ object acts as a screen for the ‘PaintCode’ UI to be displayed.   13 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   _________________________________________________________________________________________ 3. Innovative Aspects of the Work ‘FLO’ is innovative and differs in comparison to successful apps such as 'Day One', 'Live Journal' and ‘Evernote’ (note taking apps), Native Instruments ‘iMaschine’ and Propellerhead ‘Figure’ (beat making apps) in three fundamental aspects: 1. Template for music creation, storage and transfer: ‘FLO’ offers a music specific template for not only creating, but also storing and transferring ideas for further development on common DAWs such as Ableton or Pro Tools. Current beat making apps function by having the entire track produced on the app itself, then exporting it as a single bounced audio file. This only allows very limited options for further development on professional DAWs, whereas the MIDI capability of ‘FLO’ is a significant advantage for further creative development. 2. Audio to MIDI conversion: ‘FLO’ features a highly appealing Audio to MIDI conversion feature, which is not currently found in major music apps (and those which attempted it have received largely negative reviews). ‘MIDIMorphosis’ is by far the best-received iOS MIDI conversion app but is made specifically for guitar and bass only.   14 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (MIDImorphosis on the App Store) A raw guitar sound is a much purer tone than a human voice, thus there are significantly fewer pitch frequencies for the app to analyze. Furthermore, there are technological considerations regarding monophonic, homophonic and polyphonic analysis; the latter of which is also significantly easier with a pure tone such as guitar or bass. Whilst software such as Celemony’s Melodyne and Neptune’s Autotune are able to do Audio to MIDI conversion well, it is difficult to replicate their success on an iOS app due to the complexity of this technology and the audio quality of the iPhone and iPad microphone for recording raw sounds. (Celemony Melodyne’s pitch correction feature)   15 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Neptune Autotune’s pitch correction feature) Audio to MIDI conversion involves complex amplitude and spectral analysis of frequency data from an audio source in order to determine a note’s pitch, length and velocity. These processes require not only audio processing and mathematical operations, but also file management, storage and transfer, which require the combination of several programming languages. With ‘FLO’, I was required to implement Pure Data, Objective C and C++, along with their corresponding object libraries and foundations, libpd for Pure Data, Parse Foundation and AV Foundation for Xcode. As of today, ‘FLO’ is able to replicate the efficiency of Audio to MIDI conversion with as much success as ‘MIDIMorphosis’ on guitar for its monophonic capabilities. As ‘FLO’ is predominantly designed for quick ‘MIDI note taking’, homophonic and polyphonic capabilities were not my top priority. 3. MIDI note taking: Unlike ‘iMaschine’ and ‘Figure’ which only transfer audio files via USB, ‘FLO’ can transfer MIDI files of beats and melodies as well audio recordings and photos online via the Parse API. MIDI files are superior to audio files in terms of size, transfer speed and flexibility for manipulation. (Native Instrument iMaschine’s UI design)   16 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Propellerhead Figure’s UI design) Based on my research with Berklee business majors Carl and Tanya into competing apps, we found few songwriting apps that compete with ‘FLO’s ‘MIDI note taking’ and Audio to MIDI conversion features. (Survey for ‘FLO’) The survey that I conducted with my business partners confirmed the demand for these innovative features that ‘FLO’ offered. As seen in the survey responses, creating and sharing MIDI files proved to be a major aspect of compositional and collaboration process for modern day songwriters.   17 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   (Survey for ‘FLO’) _________________________________________________________________________________________ 4. New Skills Acquired I acquired many new skills for my Culminating Experience project. I had no programming experience prior to my studies at Berklee and I was required to learn several languages in order to program my iOS app: 1. MAX MSP 2. Pure Data 3. Objective C It is also worth mentioning the different capacities in which I used these programming languages. Not only did I grow familiar with these programming languages, I also developed the fundamental skills and problem solving logic for conceptualizing, designing and executing an original iOS application from scratch: 1. Implementation of Parse API into Objective C 2. Beat Quantization logic 3. Audio to MIDI conversion logic 4. Real time synthesis 5. Implementation of external objects and libraries (‘midifile’ object and ‘libpd’ library) 6. Communication between several programming languages, object libraries, frameworks and servers   18 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   Asides from the new skills I gained in programming, I also gained new skills with software such as Ableton (which I had no prior experience), that provided the initial inspiration for ‘FLO’s MIDI note-taking function. Other skills included: 1. Independent research through online forums and tutorials 2. Time management skills 3. Presentation skills 4. Collaboration skills _________________________________________________________________________________________ 5. Challenges Encountered As my Culminating Experience project involved the extensive use of programming languages, which I had no prior experience with before coming to Berklee, I faced many challenges throughout the entire process. During the initial conceptualization stages, the biggest challenge I faced was the need to be innovative. This challenge was especially difficult given the many new and innovative technologies emerging for iOS applications today. With guidance from my program director Stephen Webber, advisor Ben Hogue and visiting professors David Mash and Richard Boulanger, I was able to incorporate the defining characteristics of an innovative product into my final app. Upon reaching a focused objective for my project, the process was then complicated by our substantial coursework and my limited programming skills. Throughout the entire execution process, the steep learning curve required for improving one’s programming skills became evident and I revised my earlier ambitious vision for more specific weekly goals given the short timeline. As I developed my skills, I was then able to develop a more realistic vision for my final application. _________________________________________________________________________________________ 6. Future Ramifications for the Project In our modern digital information age, recording personal experiences and sharing them with the world via social networking is increasingly expected of us. ‘FLO’ has the potential to contribute to this 21st century   19 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   zeitgeist by allowing creative content to be recorded, stored and transferred as MIDI files online. This would open new and exciting doors for creative collaborations between ‘FLO’ users as their ideas could potentially be integrated as a feature on social networking platforms such as Facebook or Twitter via integrating the relevant API into its code. As indicated in my survey, the majority of musicians collaborate with each other via sharing files such as MIDI, audio samples and text. This shows the potential for ‘FLO’ to blossom into a larger scale project with a team of programmers and established business partners (i.e. Ableton, Pro Tools, Facebook) to develop a useful and innovative format for everyday users to share their musical notes online. Furthermore, visiting artists such as DJ Guru stated that a MIDI note-taking iOS application could become the ‘standard’ for capturing musical ideas. _________________________________________________________________________________________ 7. Conclusion In summary, ‘FLO’ is designed to provide the utmost speed and efficiency in capturing the basic essential details of a songwriter’s daily inspirations. Through its 3 features, ‘FLO’ allows users to quickly record, store and transfer musical ideas as MIDI files onto DAWs, such as Ableton and Pro Tools, for further development through the Parse server API. From here, users can import that MIDI data onto their computer to start or continue working on a bigger project such as a musical composition or track. Creating ‘FLO’ was a very challenging and fulfilling experience for me. Not only did I acquire a varied range of new skills, I was also able to open my mind to different perspectives of music technology as a developer and user. This was a new and ambitious project for me and I am thankful for all the inspiration and guidance that I received during my studies at Berklee.   20 MM  (MTI)  Berklee  College  of  Music:  Final  Documentation                                                                            Austin  Har  763777   _________________________________________________________________________________________ 7. Footnotes The following are the main sources from which I developed my programming skills, along with consultations from my supervisor Ben Hogue: Peter Brinkmann, 2012. Real-Time Audio Synthesis on Android and iOS: Making Musical Apps. O-Reilly Media Inc. www.stackoverflow.com (online question and answer forum for programmers) www.lynda.com www.youtube.com http://ipgames.wordpress.com/tutorials/writeread-data-to-plist-file/ http://www.raywenderlich.com/36341/paintcode-tutorial-dynamic-buttons https://developer.apple.com/ https://www.parse.com/ http://en.flossmanuals.net/csound/e-csound-in-ios/ https://github.com/libpd/pd-for-ios/wiki/ios _________________________________________________________________________________________ _________________________________________________________________________________________   21