Berklee College of Music SPATIAL ILLUSIONS: AN AUDIO-VISUAL EP THAT WILL ENCOMPASS VIVID, DYNAMIC PROJECTION MAPPED VISUALS Submitted in Partial Fulfillment of the Degree of Master of Music in Music Production, Technology and Innovation Supervisor: Pierce Warnecke By Li-Ying Ooi Valencia Campus, Spain July 2019 Table of Contents Abstract iii Acknowledgements iv 1. Introduction 1 2. Review of The State of Art 1 2.1 One of A Kind 1 2.2 Portable Projection Mapping 2 3. Description 3 3.1 Audio Portion 4 3.2 Visual Portion 4 4. Innovative Aspects 5 5. New Skills Acquired 6 5.1 Music Creation Skills 6 5.2 Visual Creation Skills 7 5.3 Projection Mapping 9 5.4 Kirigami 9 6. Challenges 10 7. Future Ramifications 11 7.1 Projection Mapping for Dummies 11 7.2 Marketing and Distribution 12 8. Conclusion 13 Appendix A. Scaled-down Nett of the Sculpture 14 Appendix B. Google Searches for the State of Art Review 15 Bibliography 17 ii Abstract The EP, Spatial Illusions, will be presented in the form of a traditional CD album, but instead of opening up to a compact disc, a pop-up structure will emerge when opening the CD case, and that will be the object that is to be projection mapped with some original music. The music itself will be in a hybrid genre called Melodic Dubstep. This genre was chosen because it embodies the beautiful and lush elements of genres like Trance and Future Bass, but also have the grittiness and the sprightly energy of Dubstep. The music will be accompanied with vivid, vibrant visuals to create a unique audiovisual experience which will be executed via projection mapping. This paper will not only address the different audio and visual aspects of the project, but also the challenges faced and next steps of the EP. Keywords: projection mapping, video mapping, projectors, melodic dubstep, pop-up, CD case, CD album, abstract, audiovisual. iii Acknowledgements Throughout the course of this project, I have received a great amount of support and assistance. I am truly grateful to my supervisor, Pierce Warnecke, for his guidance and encouragement, especially during times of doubt when I was uncertain about the project. I would also like to express my appreciation to Ben Cantil for mentoring me with the music aspect of the project. It was the most nerve-wrecking portion of the project and his knowledge and insight has been extremely helpful to my development as an artist. Many thanks to Devon Bryant as well for providing me with so much new knowledge on an art form that I have never encountered before, it was immensely eye-opening, pun intended. My gratitude also goes to the panel, Pablo Munguía, Pierce Warnecke, and Ben Cantil, for their time and constructive criticisms during the prototype sessions, and also Stephen Webber and Katerina Becic for their encouraging feedback during the final defense. I am also grateful to all my peers in and out of Berklee for their praise and support. Their compliments have given me confidence in my artistry, visually and musically. And finally, my parents, who have supported and provided for me all these years, for which I would not have gotten this far without them. iv 1. Introduction The uprise of streaming on digital platforms has caused the decrease of the physical means of music consumption,1 creating a dent in the music market as artists lose an extra outlet for revenue.2 Hence the idea of this unconventional EP was born. It will utilize the lowly CD case as the physical media, but with a non-traditional purpose. To be clear, this project does not aim to bring back the use of CDs, but just to revive the tangibility of holding the physical object in hand. The user will be greeted with a pleasant surprise when opening the CD case. A paper sculpture appears before them, unexpectedly replacing the CD. Its unorthodox usage will be used to advocate the art of projection mapping, making it an extremely unique CD album, and at the moment, the only one of its kind. The EP is titled Spatial Illusions due to the visual aesthetics of the projections creating light movements onto layers of mapped space, then combined with energetic music to create a unique audiovisual experience. 2. Review of the State of the Art Spatial Illusions is not just another audiovisual EP, but a way of innovating old media with new objectives. This section will assess the individuality of the EP’s physical nature and the technology of projection mapping in a cozier setting. 1 Meira Gebel, “The Walkman Just Turned 40 – Here’s How Listening to Music Has Changed Over the Years,” Business Insider, June 1, 2019, https://www.businessinsider.com/history-listening-to-music-recorded-walkman2019-6?IR=T. 2 Tim Ingham, “The Album is in Deep Trouble – and the Music Business Probably Can't Save it,” Rolling Stone, November 9, 2018, https://www.rollingstone.com/music/music-features/the-album-is-in-deep-trouble-and-themusic-business-probably-cant-save-it-753795/. 1 2.1 One of A Kind Presently, it seems like there isn’t any other body of work that is anywhere similar to Spatial Illusions. Google searches like ‘Projection Map Album’, ‘Projection Map CD Album’, ‘Projection Mapping CD Case’, and ‘Pop-up CD Case’ do not yield any relevant results. Most results refer to projection mapping as an installation or an event. Some also refer to projection mapping as a geographical item due to the word ‘map’. Google searches about repurposing the CD case only show concepts that destruct the CD case instead of embracing it. It is worthwhile mentioning the first pressing of Michael Jackson’s Dangerous, which was a collector’s edition album with some pop up artwork like a children’s pop up story book, but it does not contain any animations or moving parts and is purely for display3. Therefore at this present time, it’s safe to call Spatial Illusions a one of a kind projection-mapped album. The art of Kirigami was used to construct the physical structure built into the CD case. Unlike Origami, Kirigami was used because it involves more than just folding of the paper. In addition to folding, Kirigami also involves more cutting of the paper that will result with a 3D paper creation.4 But this is where most of it ends, a final product that is just an intelligently cut and folded piece of paper in a card or book. By squeezing it into a CD case to compress it and using it for projection mapping with music, it creates an innovative piece of art using projection mapping to highlight its beauty and design. 3 “Michael Jackson - Dangerous Collector's Edition First Printing 1991 Unboxing 4K | MJ Unboxing,” October 8, 2017, YouTube video, 2:33, https://www.youtube.com/watch?v=fCeQqwDh0qE. 4 Kelly Richman-Abdou, “Kirigami: The Ancient Art of Paper Cutting and How Artists Are Keeping It Alive,” My Modern Met, April 22, 2019, https://mymodernmet.com/kirigami/. 2 2.2 Portable Projection Mapping Most of the projection mapped pieces out there are usually large in scale on stage, in a festival, or in a live show. So in order to enjoy a projection mapped installation in person, one would usually have to attend an event or go to a large venue to enjoy this art. Whilst that is great, it is not common practice to have projection mapped art in a more intimate setting. There is now an increasing number of affordable small and portable projectors on the market today. Which means it is possible to enjoy the art of projection mapping in a smaller, more intimate setting. Major brands like Sony and LG not only have projectors for professional use, but they also have a wide range of projectors for home use, big and small. The Sony MP-CD1 Mobile Projector 5 and several of the Minibeams from LG 6 are palm-sized and can project from one meter to three meters. Other shapes and sizes like Anker’s Nebula Capsule 7 in the shape of a soda can and Merlin Digital’s PocketBeam Cube 2 8 in the shape of a small cube can project a similar range which should be sufficient enough to projection map a small object. Therefore, there are opportunities to enjoy projection mapping art on a consumer level, rather than just using the projector to watch movies or play games. Hoffman (2019) compiled a list of mini projectors in the market indicating that there is a demand for compact projectors.9 Therefore, with a portable projector and a portable sculpture, no fancy technology is required for an average consumer to participate in projection mapping. 5 “MP-CD1 Mobile Projector,” Sony.com, accessed 4 July, 2019, https://www.sony.com/electronics/projector/mp-cd1#product_details_default. 6 “LG PW 1500G”, Lg.com, accessed 4 July 2019, https://www.lg.com/pa_en/projectors/lg-PW1500G. 7 “Capsule Projector ,”Anker.com, accessed 4 July 2019, https://www.anker.com/store/capsuleprojector/D4111111. 8 “PocketBeam Cube 2”, Merlin-Digital.com, accessed 4 July, 2019, https://merlin-digital.com/pocketbeamcube2.html. 9 Tony Hoffman, “The Best Portable Projectors for 2019.” PC Mag, 17 June, 2019, https://uk.pcmag.com/projectors/65529/the-best-portable-projectors. 3 3. Description Spatial Illusions not only aims to create visual stimulation, but auditory invigoration as well with the melding of the audio and visual. Both audio and visuals have their own role, but collectively, they create the unique audiovisual experience to really captivate the consumer’s eye. 3.1 Audio Portion Melodic Dubstep is a genre that combines the pretty and the ugly together. It also tends to be quite emotional, euphoric, and more musically pleasing compared to pure Dubstep.10 The musical goal for this EP was to create beautiful sounding introductions and breakdowns with rich melodies and lush pads, while the drops and choruses have harsh and gritty elements but still maintaining some sort of melodic sequence. Thus, three tracks were born with this concept in mind, blending together calm and chaos to sound as a cohesive musical piece. 3.2 Visual Portion No stock imagery is used in the creation of this audiovisual EP. Everything has been created from scratch, designed to fit within the confinements of the mapped sculpture. The projected visuals will be of abstract style with complicated patterns and will be animated to create dynamic movement within the piece. Although the colours will be increasingly saturated, it is still necessary because it will guarantee a bright and vibrant output from the projector. The visuals will be then tied to the music. For example, a pumping animation will be connected to a pulsing kick drum in the song, and gradual 10 Phyllis Feng, “8 Emotion-Filled Melodic Dubstep Songs that will Suck You into the Genre,” Affinity, October 24, 2018, http://culture.affinitymagazine.us/8-emotion-filled-melodic-dubstep-songs-that-will-suck-you-intothe-genre/. 4 fades for slower sections to emphasize a more euphoric ambience. All these visuals will take place on the paper structure constructed using simple Kirigami techniques. 11 As the sculpture will be made simple due to size constraints, creating three versions of the same design but at different scales would make it more interesting. Replicating the sculpture three times at different scales will create three layers of different projections. This creates some opportunities for some unique animated visuals. Furthermore, it’s a unique experience opening up a CD case and have something totally unexpected fan out before the person when the three layers rise up in front of the audience. The different layers will interact with each other using different colours and animations to create a unique audiovisual performance which can be seen below in figure 1. Figure 1. Example of the abstract visuals on the different layers of Kirigami art. 11 “Simple Kirigami Tutorial,” May 12, 2016, YouTube video, 3:55, https://www.youtube.com/watch?time_continue=213&v=7VoP-Q6UESw. 5 4. Innovative Aspects This is innovative in two ways. Firstly, it can be an interactive way for people to participate and enjoy the EP, as opposed to just putting the CD into a player and pressing play. It is a different way to consume music, a treat for the eyes and ears. The enticing visuals will complement the music, creating an audiovisual experience which is so much more than just a listening activity. Secondly, it can revive the purchase of albums again as opposed to solely using digital downloads or streaming methods to consume music and can possibly reopen an outlet of revenue for artists. The EP will not only contain original music work, but original artwork too, combining to art mediums into one cohesive art piece. 5. New Skills Acquired A variety of new skills had to be acquired in order to create this EP. These skills have been learnt during classes, extra office hours inquiries, independent research, and through many mistakes made along the way. 5.1 Music Creation Skills This was the first time creating a musical body of work from the very bottom. Not using any loops, not doing a remix, not collaborating; not using someone else’s idea as a basis has definitely been challenging. All three tracks were created solely in Ableton Live and consists of many original sound design elements from soft synths like Xfer’s Serum and Native Instrument’s Massive. These skills were acquired in both of Ben Cantil’s classes. Other skills like mixing and composition skills were also acquired in those classes, but was most beneficial outside of class as implementation of the techniques learnt in class were then applied to the compositions. Other technical musical skills learnt was also using iZotope’s Insight 2 for checking the 6 LUFS (Loudness Unit Full Scale)12, and Vocal Synth for altering the timbre of the vocals in the last track. Taking the DJ class with Nacho Marco was more helpful than anticipated. The exploration of the Melodic Dubstep genre was done mostly trying to execute DJ sets in the genre. Discovering how the intros and outros were so alluring and mystical, dissecting how thay transitions into hard and heavy drops, and just balancing the clean and dirty so it didn’t sound bipolar. It was also important that there were some emotional elements to evoke the listener with some high impact feelings. All the assigned coursework for that class became research when writing the tracks for the EP. 5.2 Visual Creation Skills Most of the visuals were made in a variety of Adobe software. They start as a single photo, then manipulated in Adobe Photoshop to create a beautiful piece of symmetrical abstract work. This is then brought into Adobe After Effects by blending two layers of the same abstract pattern but rotating one to the left, and the other layer rotated to the right at different speeds. The aesthetic results will vary depending on the colour, blend mode, scale, and frequency of rotation. This results in a piece of trippy animation that can be used for the projection mapping part. However, these are skills have been acquired over the years prior to coming to Berklee and are not new skills acquired. 12 Craig Anderton, “What is LUFS, and Why Should I Care?”, Sweetwater Sound, November 7, 2018, https://www.sweetwater.com/insync/what-is-lufs-and-why-should-i-care/. 7 New skills learnt in Adobe After Effects were using learning to create line traces. This type of animation creates outlines that progress over a series of time and can be done by controlling any of the stroke offset parameters on a shape layer. But another technique to create more exhilarating line traces was to use a plug-in called Saber by Video Copilot. This plug-in is able to simulate a variety of different designs like fire, electricity, and many more, to outline the desired shape. This plug-in was used heavily as it was able to generate a wide array of different designs by just duplicating and varying by altering the parameters like colour, glow, etc. Another new visual skill was learning to use Cycling 74’s Max to generate visual content, taught by Pierce Warnecke in the Non-linear Structures for Real Time Media class. A few different types of abstract visual animations were generated from using V-Synth and Syphon Recorder. These visuals are more random and less symmetrical, but are still able to achieve desirable results of charismatic movement and colour. A mockup of the different layers was needed in order to be able to visualize how the different layers of projections will interact with each other. This was done in Adobe After Effects, by having one main composition with the three different layers, and three sub-compositions representing each layer. As seen in figure 2, the composition labeled ‘Demo Comp’ was the main composition to see all three layers in action. Highlighted in red are then divided and animated individually in the layers ‘Big’, ‘Mid’, and ‘Small’ each representing a layer in ‘Demo Comp’. Any changes made in the individual compositions will be reflected in the main composition as the animation process progresses. 8 Figure 2. Demonstrating the visual animation process on the different layers of the sculpture. 5.3 Projection Mapping To composite the layers together, the individual were exported as individual files with a .DXV format. Then brought into Resolume Arena 6 which is also divided into three individual layers. The Advanced Output section is the most crucial part to the projection mapping process. With the projector connected and in the Advanced Output using the Polygon Tool, the sculpture is then mapped out, each layer will have it’s own mapping. Each layer/ mapping in the Advanced Output section will correspond to a layer in the Resolume session. This will result into the different layers acting on their own, but play together in sync with the music. This is a new skill acquired in Devon Bryant’s Live Video class. 9 5.4 Kirigami Building the sculpture was a heavy trial and error journey. Testing out different shapes, size, distance between the layers, and many other factors that had to be considered. The most important one was the stability of the whole sculpture, and how to compress it into the CD case. It had to be tested multiple times with different dimensions with different paper densities. Figure 3. The amount of experimentation needed to test effectiveness of the sculpture. 6. Challenges Most of the expected challenges were mostly of the technical sort. It was expected that the first encounter with projection mapping was going to be difficult and unfamiliar. Deciding what kind of sculpture to squeeze into the CD case was also a challenge due to size 10 constraints. Even during the mapping process, if the projector moved ever so slightly, the entire mapping would shift and would become inaccurate. Having to do the mapping in a short amount of time made presentation time rather stressful as well. But it was discovered that even though it can be easily destructible, it is actually a very forgiving art due to the bright, vibrant colours and vigorous movements of the animations. Abstract patterns were also chosen to create more flexible imagery so it still manages to captivate the audience even if the projections escape the confinements of the mapping. Most of all, each component required several trials in order to figure out the best part that worked the most efficiently. For example, learning and practicing mapping with Resolume, creating multiple versions of the visuals to figure out which type of colour combination, animation movement, and music interaction worked best. Patience was also tested during the learning process when a million mistakes were made and waiting for the animations to render. The unexpected challenges were mainly on a mental and emotional level. Balancing too many classes at once, saying yes to too many peer projects, maintaining a healthy lifestyle, uncertainty of the project’s outcome, doubting the success of the project, and paranoia that the project would receive negative feedback, etc. These obstacles affected presentation time with a lot of nerves and anxiety even with a lot of preparation. The best way to overcome this is just to be confident personally and with the project as that helps to eliminate the doubt and uncertainties. 7. Future Ramifications This first stage of the project was to mainly focus on the content and exploring the art, musically and visually, the next steps would be how to distribute the EP as a final product. As 11 one of the current issues is that projection mapping is not a user-friendly process, two aspects are pivotal in order to achieve feasibility for commercial consumption. 7.1 Projection Mapping for Dummies Projection mapping can be a tedious process and requires a certain level of technical skills in order for it to be executed effectively.13 The goal would be to make the projection mapping process simple enough for any average consumer to accomplish the mapping process. One possibility to tackle this is to use Augmented Reality technology. By using AR technology to map the sculpture, the user can use their mobile devices to hover over the sculpture via a mobile application and it will then trigger the music and project the visuals onto the sculpture when the target image has been detected. For the detection of the sculpture, a mobile application can be developed with platforms like Vuforia and Unity 3D.14 But the actual mapping after detection within the application will still need to be further researched. This method will make the EP more accessible to consumers and would not require any advanced technical skills to enjoy the audiovisual experience. Furthermore, this will be an interactive component for the consumers which can be educational about the art of projection mapping. Another alternative would be to have the EP be presented as an art installation. The consumer would purchase the EP as mini installation installed at a location of their choice. The installation would be pre-mapped, so the user just has to turn the installation on in order to enjoy the EP. This would make the EP more exclusive and could have a higher price value due to the installation of the installation. Furthermore, the user would 13 “Projection Mapping Series Part Three: Pros, Cons, and Alternatives to Projection Mapping,” Ascend, 9 October, 2018, https://ascendstudios.com/projection-mapping-pros-cons-alternatives/. 14 “How To: Augmented Reality App Tutorial for Beginners with Vuforia and Unity 3D,” Posted by Matthew Hallberg, July 3, 2016, YouTube video, 19:22, https://www.youtube.com/watch?v=uXNjNcqW4kY. 12 not need to worry about the mapping process at all which could be appealing to certain audiences. 7.2 Marketing and Distribution One approach will be to partner with a projector company that manufactures miniprojectors and sell the EP with a mini-projector included as a set. This could work as a gimmick as it is unusual to purchase music with video projectors. Other promotional gimmicks could also be to include a vinyl version to offer a bigger sculpture as an option. The vinyl version will not only include a larger sculpture, but also the mini-projector as well as a copy of a pressed vinyl. Getting signed with a label could also help with the funding and budget of the EP and would alleviate all the projector costs or installation costs. It could also help with the awareness of the EP if marketed through the label. Crowdfunding could also be another alternative that can be explored. The benefits of this is that there will be more interaction with the interested audiences and also that there will be more freedom as there will not be any contractual obligations. 15 Even though tangibility plays a huge role in this project, having the music available on all streaming platforms is also vital as streaming is still one of the most common methods of music consumption.16 This will be helpful in marketing and 15 Haakon André Søttar Hagen, “Crowdfunding Music: A Look at Crowdfunding as an Alternative Source of Funding for Independent Creators Within the Recorded Music Industry,” Master’s thesis, University of Agder, 2017, https://pdfs.semanticscholar.org/a590/d114f9f6cb5d4775e1672349850f42365537.pdf. 16 Chris Smith, “Download Music vs Streaming Music: Which is best for you?” BT, 28 February, 2019, https://home.bt.com/tech-gadgets/phones-tablets/downloading-music-vs-streaming-music-which-is-best-foryou-11363974732804. 13 promoting the EP as it could even encourage audiences to purchase the physical EP if they enjoyed the music first. 8. Conclusion This unusual EP may look like an outdated piece of media on the outside, but once opened, the user is greeted with a pleasant, unexpected surprise. A different spin on the old CD case can bring back the love we once had for the CD but in an innovative way with a different purpose. The art of projection mapping should also be embraced as it has such a big impact and has a very strong ‘wow’ effect. These initial stages focuses on the content itself, the next step would be how to make projection mapping simple enough for an average consumer and mass produce the final deliverable. Even though there is emphasis on the physical product, streaming is still a popular and convenient method of distributing music and is still the preferred way of music consumption. Therefore, having the online platforms could create awareness of the existence of the physical EP and encourage people to purchase it after listening to the EP online, opening an extra outlet for revenue for artists. But most importantly, the audience will have an interesting audiovisual experience whilst enjoying this activity of consuming music. 14 Appendix A. Scaled-down Nett of the Sculpture 15 Appendix B. Google Searches for the State of Art Review 16 17 Bibliography André Søttar Hagen, Haakon. “Crowdfunding Music: A Look at Crowdfunding as an Alternative Source of Funding for Independent Creators Within the Recorded Music Industry.” Master’s thesis, University of Agder, 2017. https://pdfs.semanticscholar.org/a590/d114f9f6cb5d4775e1672349850f42365537.pdf. Anderton, Craig. “What is LUFS, and Why Should I Care?”, Sweetwater Sound, November 7, 2018. https://www.sweetwater.com/insync/what-is-lufs-and-why-should-i-care/. Feng, Phyllis. “8 Emotion-Filled Melodic Dubstep Songs that will Suck You into the Genre.” Affinity, October 24, 2018. http://culture.affinitymagazine.us/8-emotion-filled-melodicdubstep-songs-that-will-suck-you-into-the-genre/. Gebel, Meira. “The Walkman Just Turned 40 – Here’s How Listening to Music Has Changed Over the Years.” Business Insider, June 1, 2019. https://www.businessinsider.com/historylistening-to-music-recorded-walkman-2019-6?IR=T. Hoffman, Tony. “The Best Portable Projectors for 2019.” PC Mag, 17 June, 2019. https://uk.pcmag.com/projectors/65529/the-best-portable-projectors. “How To: Augmented Reality App Tutorial for Beginners with Vuforia and Unity 3D.” Posted by Matthew Hallberg, July 3, 2016, YouTube video, 19:22, https://www.youtube.com/watch?v=uXNjNcqW4kY. Ingham, Tim. “The Album is in Deep Trouble – and the Music Business Probably Can't Save it.” Rolling Stone, November 9, 2018. https://www.rollingstone.com/music/musicfeatures/the-album-is-in-deep-trouble-and-the-music-business-probably-cant-save-it-753795/. “Michael Jackson - Dangerous Collector's Edition First Printing 1991 Unboxing 4K | MJ Unboxing.” October 8, 2017, YouTube video, 2:33. https://www.youtube.com/watch?v=fCeQqwDh0qE. “Projection Mapping Series Part Three: Pros, Cons, and Alternatives to Projection Mapping.” Ascend, 9 October, 2018. https://ascendstudios.com/projection-mapping-pros-consalternatives/. Richman-Abdou, Kelly. “Kirigami: The Ancient Art of Paper Cutting and How Artists Are Keeping It Alive.” My Modern Met, April 22, 2019. https://mymodernmet.com/kirigami/. Smith, Chris. “Download Music vs Streaming Music: Which is best for you?” BT, 28 February, 2019. https://home.bt.com/tech-gadgets/phones-tablets/downloading-music-vsstreaming-music-which-is-best-for-you-11363974732804. “Simple Kirigami Tutorial.” May 12, 2016, YouTube video, 3:55. https://www.youtube.com/watch?time_continue=213&v=7VoP-Q6UESw. 18