SongTree

SongTree Visual Interface

Inspiration

Generative machine learning tools are one possible route to creating ambient and meditative compositions. This project was inspired by the idea of creating a link to a natural de-stressing environment from within the browser, and from our curiosity about one of the more peaceful natural sounds in existence: the calls of birds.

Birdsong is both aesthetic and functional, fundamentally similar and yet very distinct from human melodies. The work of music theorists has addressed the structure and phrasing of bird-calls and melodies. Physical modeling approaches have described the mechanisms and acoustic properties of birdsong and allowed for some imitative synthesis. More recently, machine learning approaches have been turned to the problem of bird species identification. But what might machine learning have to offer in terms of generating bird-like melodies?

What it does

To explore these questions, we attempted to build an interactive and meditative browser experience leveraging magenta to generate variations on bird-calls. Unfortunately we were unable to complete a fully working app in the allotted time. However the individual components and prototypes that we were able to complete are showing promise!

The video demo shows the visual environment running in the browser, with samples of ambience and re-synthesized bird calls from our "Bird MIDI" dataset.

How I built it

The visuals were built with React.js using a geometric and algorithmic approach to generate minimalist and gentle interpretations of a natural scene, centering on a tree.

The audio framework was built with Tone.js, incorporating sample playback to provide a peaceful and musical/natural soundscape, and FM synthesis to realize the bird-calls.

In order to coax birdlike songs from Magenta's MusicVAE models, we created our own database of "Bird MIDI," using a specially tuned FFT to find the dominant frequencies in clips of birdsong from Kaggle's British Birdsong Dataset and converting these to their nearest MIDI note number.

However we ran into difficulty attempting to train our own model using MusicVAE, MusicRNN, and MidiME - these efforts are ongoing.

Challenges I ran into

Navigating the Magenta documentation; organizing our team effort; and accounting for different levels of familiarity with JS and Python.

Accomplishments that I'm proud of

The problem has been posed before: what is bird-song's relationship to human musical representation? However the practical possibility of being able to search the latent space of typical melodies for similar melodies to those called by the bird; to interpolate between bird and human melodic sequences, and much more, now appear tantalizing close with the right combination of machine learning and creative transcription of the birdsong itself.

What I learned

Our whole team has become familiar with new aspects of web technology, machine learning, and Google's magenta.

What's next for SongTree

SongTree needs to be finished and given a home to live on the internet! Even though we were not able to complete our project fully in the time frame, we have high hopes to still do so.