Moving Music

Inspiration

We were very interested in computer-generated music, however sometimes generating music with neural nets results in long passages of repetitive chords that the net has difficulty escaping out of. To avoid this scenario as well as make the entire music generating process more enjoyable and active, we decided to create a program that would generate music using movement as a live seeding mechanism to avoid the scenario of the net stagnating, as well as providing some extra fun to the whole experience.

What it does

Moving Music tracks a specific object and its movements. It translates both the positioning of the object as well its velocity and converts these values into seeds for the music generating RNN. We pass these seeds into three RNN's for three separate instruments, with each seed slightly modified. The RNN's are then run in a loop, constantly checking for updates in the seeds while generating new music.

How we built it

We built Moving Music by first identifying an object using colors with an opencv module. We then tracked that object's center and velocities, and passed those values as parameters to our music-generating modules. We used magenta's open source RNN library to generate our music. We fixed some of the parameters of the net and let others be dictated by our motion tracking. Putting this together we got a product that generated music by translating position and velocity into complexity, pitch, and tempo.

Challenges we ran into

The main difficulty we had was trying to tune the program's output to actually sound like music. We needed to select a range for each parameter small enough such that music produced within that range still sounded like music, while also large enough that moving the object around produced a notable change in the music. Ultimately we solved the challenge not by tweaking the parameters, which were very stubborn in what they could produce, and instead changed the structure of our music generation to rely on multiple instruments each producing notes at a slower rate, and relying on an "ensemble" of sorts to produce the music, rather than having a solo instrument.

Accomplishments that we're proud of

We're most proud of our hack on magenta's Performance RNN that allows us to do pseudo-live updates. Performance RNN usually produces a music clip of set size, and follows a strict procedure of: receive parameters, process, return audio file. This obviously wasn't an option since we wanted our music to be able to update live, and so we had to create an elaborate wrapper function that would constantly keep multiple performance rnn threads queued up and play them at the right time and in the right order to give the illusion of live updates. We were particularly proud of this simply because of how many moving parts were involved and how critical timing is to music, so to have accomplished it and produced music that still sounded like music was truly satisfying.

What we learned

As cliché as it sounds the most important lesson we've learned once again is not to cut corners while reading documentation. It's probably something that nearly everyone does, especially in a time-sensitive situation, but it's not worth the hours it costs later on trying to find the right function or object attributes, pouring through pages upon pages of documentation and finally discovering that everything you needed was in one of the first docs you glanced over.

What's next for Moving Music

I think the next step for Moving Music is to speed up its processing and make its movement recognition more versatile. If we can do both these things then it's exciting to imagine what the program could do. Dancers wouldn't have to select music to perform to, the music would generate itself to suit the dancers' needs. Perhaps we could replace motion with audio, imagine a tv show, audio book, or video game where the music generates itself to suit the mood, and where the same song will never be played twice! Regardless of what we do with Moving Music, we believe that the AI is too convenient a tool to not be implemented in every frivolous idea, and so we'll be sure to continue hacking and moving forward to the sounds of our own music.