I am mostly inspired by the works of Tero Parviainen and Monica Dinculescu.
The idea for my project came from the architecture of MusicVAE and how new samples are requested from it, similar to the visual stack of sequences in Infinite Drums.
What it does
Infinite Drums samples drum sequences using Magenta.js from the MusicVAE model. To give the visual impression of infinity, a stack of drum sequences is shown in 3D. The user can interact with the project in two ways:
- Via "Next" button: Pressing the "Next" button plays back a new drum sequence from MusicVAE and visualises the pattern
- Via mouse drag & zoom: The user can rotate the 3D scene and zoom in and out
How I built it
The drum sequences are generated by Magenta.js using the MusicVAE model with "drums_2bar_lokl_small" checkpoints. For music playback I use Tone.js with custom audio samples, played back using a Tone.Players instance. To glue the sounds together I chain the output of the players into an equalizer, as well as a reverb. For the visualisation I use cables, a visual programming editor for WebGL. In cables I render the current sequence of drums as spheres. To visualise the infinity of the latent space I also render some more (19.200) spheres in the z-direction.
Challenges I ran into
I ran into various challenges, which I have partly solved. Some issues need to be addressed after the hackathon.
- Finding out which note pitches belong to which instrument. I plan on publishing a small library to make this task easier in the future.
- Keeping audio and video in sync: I followed the Tone.js recommendations, but it seems like I did something wrong in drawing synchronous to the audio
- Using a bundler: Currently cables is not compatible with ES6 or CommonJS module bundlers. To save bandwidth I have to find a better solution how to bundle all code and minimise the assets.
- Loading indicators and handling: It should be indicated when assets are being loaded and when it is ready to be used
- Sometimes drum samples seem to only contain one bar, so half of the sequence is empty. I should filter these out or repeat them to get rid of the silence.
- Animations between states: Initially I planned on animating drum sequences from the infinite stack to the current sequence being played. There are various visual tweaks which would improve the overall project, but would require more time.
Accomplishments that I'm proud of
- Playing music from a Neural Network. I wanted to build something using Magenta.js since I first read about it. Now I finally did :)
- Finishing an entry
- Visual appearance
What I learned
- How to get samples out of a MusicVAE model
- How to map note pitches to instrument names (for example "Kick Drum")
- How to display a lot of spheres in cables using WebGL instancing
What's next for Infinite Drums
- There are a lot of possible performance improvements
- Better visual + audio sync
- Mobile optimisation
- Audio samples: Currently I am using wav files. I could save bandwidth by using mp3 / ogg (for Firefox). Also I would like to check out sound fonts.
- Bundling: Find a way to use a module bundler with cables
- Try to find better samples
- Add option for MIDI out, so the project could be used together with other audio tools