Expressiva

An AI environment developed for music producers. Featuring Allegretta, a Bidirectional LSTM raw-to-MIDI audio converter, and Melodica (coming soon), an Attention RNN music generator.

Take a look at Expressiva: http://expressiva.online/

Why?

Music writers often get the best ideas in their heads, but have trouble implementing it into software. Sometimes, the best ideas come from improvisation which can’t be played again, and other times, artists simply have no idea where to begin on their piece. Finding and inputting notes by ear from raw audio into a DAW is tedious, so producers prefer easy-to-manipulate digital files like MIDI. However, MIDI files are limited to electric instruments, and sometimes producers and composers have limited instrumental skill. We want to eradicate these blockades to music production with Expressiva.

How does Expressiva help?

Expressiva offers a service, Allegretta, to convert raw audio into MIDI for importing into a DAW workspace. Powered by Magenta and Tensorflow, Allegretta uses a Bidirectional LSTM neural network that is trained on polyphonic piano concertos to transcribe audio to MIDI. Don’t have an instrument or ready-made recording on hand? Allegretta also supports recording in real-time to catch any sparks of musical ingenuity live. Allegretta merges that barrier between raw audio inspiration and musical production.

We plan to integrate another service, Melodica, to further nurture creativity and to address music writer's block. Supported by Web Midi API and Audiokeys, Melodica intelligently generates melodies and harmonies while you play on a computer keyboard or a MIDI controller. Melodica is powered with Magenta, Tensorflow, and Tone.js and features an Attention-RNN network to predict and auto-complete your melodies as you play.

How do I use Expressiva?

Click the upload or record button to load an audio file into Allegretta. After a few seconds, Allegretta will present a midi file. You can then use the player to listen to the file or click the save button to download the file. The downloaded file is compatible with any DAW of your choosing.

How did we build Expressiva?

We used the Magenta.js, a JavaScript API geared towards machine learning for music, for both Allegretta and Melodica. We used Tonal.js, a music theory library, to convert note pitches to midi values in Allegretta. In Melodica, we used an Attention RNN architecture. We trained both models on the Magenta Maestro dataset, a dataset comprised of 200 raw MIDI and audio file pairings from the last ten years of the International Piano-e Competition. The training was done in-browser with Tensorflow.

Challenges

This is our first time working with deep learning - we've never touched neural networks before. We're pretty new at hacking too, so we definitely ran into trouble along the way. We didn't even know how RNNs worked before Creatica! We also tried to pick up new frameworks like Flask, Tensorflow, and Magenta.

The timeframe definitely challenged us - training neural networks takes hours. We scrambled to deploy Allegretta, but we're proud of how it turned out. Melodica took a long time to train, so we aren't ready to display it on the Expressiva website quite yet. Demos of both Allegretta and Melodica are in the video :)

We've never used domain.com before Creatica, so we struggled with connecting the domain. We stumbled over every unfamiliar task - building and training neural networks, developing the website, deploying the website on localhost, connecting a custom domain, and many more - but we refused to fall! Despite these challenges, we had lots of fun developing Expressiva! We hope you enjoy.

Accomplishments

Learning! This weekend was full of brutal challenges. We're proud of ourselves for staying resilient and motivated to present Expressiva.

Next Steps

Our immediate next step would be to deploy Melodica on Expressiva and add a MIDI file download component to Melodica from the session. We plan to add a youtube URL input to Allegretta, so that artists can link samples they find online. We're also looking at adding more features like implementing a cloud-based DAW in the browser to ensure direct transfer and manipulation of MIDI files from Allegretta and Melodica without download. With Allegretta, Melodica, and a DAW, Expressiva will truly be a one-stop haven for music producers and composers.

Tracks/sponsor prizes: Best Far Out Hack, Best Impact Hack, Rising Tech Hack, Best Domain Registered with Domain.com

Developed by musicians, for musicians <3

Built With

Share this project:

Updates