- A music tool anyone can use and have fun with
- Music creation made communal
- Allow professional producers to go beyond the capabilities of traditional music controllers and involve their audience in the creative process.
What it does
Ctrl^3 (Control Cubed) connects the Merge Holocube to Ableton Live. It allows producers to control software synthesizers, audio filters and light effects in up to five dimensions, pushing Ableton users to a new level of musical creativity. It allows them to control way more parameters than you could ever do before with just one hand - including visual and interactive elements. We just scratched on the surface of its possibilities, especially regarding audience involvement. Just think about building these cubes in the size of a beachball or even 3 meters wide - and throw a couple of them then right into your audience to let them play with them.
How we built it
C^3 was built with Merge Holocube, Ableton Live, Max4Live, Unity, and NodeJS. We used Unity for the user interaction and augmented reality. With NodeJS we built a server that allows Unity to communicate with a WebSockets server inside Ableton. And in Ableton would could process the data sent by the cube in VR to synthesize and manipulate music (and potentially also other media like video and lighting) in realtime.
Challenges we ran into
- Tracking an object in AR space
- Finding a way to communicate with Ableton Live in real-time
- Calibrate parameter manipulation in a way that really results in a "musical experience"
- Getting Unity to build on Android VR
Accomplishments that we're proud of
A producer (David Block) and a professional bass player (Hash from Thievery Corporation) dropped by and were literally blown away by the possibilities that new technology provides.
And: It just works. We started the project as a learning experience, expecting a failure, since the task seemed too monumental. But we made it there :-)
What we learned
We gained insight into how Ableton Live is used by professional producers and the challenges they face. These learning occurred after we decided we wanted to enable real-time, live manipulation of music, and to accomplish this connected HoloCube movements with Ableton Live as well as Amper Music.
Once we could manipulate both these tools, we had to figure out what were the right music manipulations and physical motions that would actually be useful for a producer. Being relative Ableton noobs, we reached out to David Block and learned a lot about how live producers manipulate their tracks using Ableton both in the studio and during live performances. Actually, we changed our initial concept a bit based on his feedback and focused on the idea of using the HoloCube as a synthesis controller instead of "just" launching prefabricated "live clips" and "scenes".
Once challenge they face is that it is difficult to control multiple Ableton Filters at the same time, and often times producers have to write scripts that will automate the control of filters if they want them to shift at the same time. Using the HoloLens in a 3-D space, we could use rotational position of the cube as well as the X, Y and Z coordinate position to manipulate at least 5 filters as well, and at the same time offer an amazing visualization and interactive elements with an audience.
What's next for Ctrl^3
Extend the platform so that it can be adapted to your own production needs without developer experiences, e.g. by supporting easy MIDI mapping of parameters to the controller.