Recurrent Sands

• 

  Recurrent Sands in action (1)

• 

  Recurrent Sands in action (2)

Recurrent Sands is intended as a flexible tool of sonic manipulation. It has two primary components: a granular synthesizer and a melody generator utilizing machine learning. In creating the granular synthesizer and its controls, care was taken to make the instrument user-friendly, but also non-prescriptive. The granular parts can be shaped into tonal relationships or regular meter, but they can also be left free. The melody generator creates a symbolic melody, and then plays two parts in parallel: a digital synthesizer, and the correlated pitch-shifted granular part. Either or both of these parts can be overdubbed into their own buffers, which can then be treated as a source for further granular synthesis manipulation.

The instrument uses the Nexus-UI library for its user controls interface, and it uses P5.js to visualize the current buffer and its loop constraints. The digital synthesizers and their effects are created using Tone.js, with some direct recourse to the underlying Web Audio API. The melody generation is done using Magenta.js's MusicRNN implementation of their recurrent neural network model, as well as one of their pre-trained models. The pre-trained model is far more inclined toward both tonal melody and 4/4 meter than the rest of this instrument, and a longer-term goal would be to train a more flexible model. Using the transcribed MIDI (or note sequences) from a large set of recorded improvisations could be an interesting data set with which to train a new model, in an extension of work done by George Lewis and Pauline Oliveros.

A significant challenge relates to pitch detection. I would like this instrument to be able to perform pitch detection on the sample buffers in real time. This would allow those pitches to be used as the seed for melody generation, creating a more cohesive instrument that could be capable of sustaining coherent harmonic movement over longer periods of time, or to stay within a single harmony with substantial internal sonic movement. I tried many tools to do this - the most promising is ML5.js's PitchDetection, as it runs very well in real time. However, it's limited to performing pitch detection on a live input source, and it has issues with the current build of Tone.js (I believe this is because of its use of the deprecated Web Audio method "createScriptProcessor", though I'm not positive). I was unsuccessful at debugging the interlibrary issues, but I would like to work on contributing to the ML5.js library to both fix this bug and make the class more versatile, allowing it to perform either real-time pitch detection, or offline batch rendering of buffers. The pitchfinder repository showed a lot of promise, as it has this flexibility of use, but it was far too slow to use in this instrument, causing audio dropouts and slowing down the other real-time audio processing.

The issues with processing speed in pitch detection relate to my other recurring challenge - providing visual feedback to the instrument's player without causing dropouts or glitches in the real-time audio. This was most present in the P5.js canvas drawing of the buffer and its loop constraints. Performing this at a typical frame rate caused audio issues. I slowed the frame rate and made the buffer drawing less responsive, as the audio playback is most important, but this tradeoff seems unavoidable given that Javascript does not support multi-threading. Even with an asynchronous call, at some point the drawing will need to happen, so how can it avoid any potential issues with real-time processing? For a similar reason, right now these buffers are in .ogg format - I would prefer higher-resolution audio, but given that the instrument is using eight buffers and processing more than a dozen audio events per second in some cases - and that this is all happening in real-time in the browser - using the lower-resolution audio seems like a reasonable tradeoff.

I'm proud of the depth of the sonic creativity possible with this instrument, and I plan to use it as a personal tool for my own compositions. The possibilities of the recombination of sonic materials in the evolution of soundscapes is a useful toolset for film composing work, or for free play. I hope others are interested in exploring its possibilities as well. The instrument's openness was very much an intentional choice, allowing for the development of chaotic and atonal sound worlds, as well as the ability to return to consonance and simplicity through a pruning of its parts.

I learned a great deal building the instrument. Most of my audio programming to this point has been done in Supercollider and Max/MSP, so this was a chance to deepen my knowledge of the Web Audio API, as well as the fantastic libraries that are utilized in this project. I've never used CSS or HTML for anything substantial before, so the crafting of the interface was also a nice learning experience. It's still quite simple, and the questions surrounding user interaction will be ones that I continue to ponder and address. This was a valuable opportunity to build a browser-based instrument with the support of the Gray Area and Magenta communities, and to learn from the wonderful workshops that were presented as part of this series.

Built With

• javascript
• magentajs
• nexus-ui
• p5js
• tensorflowjs
• tonejs