Melodify

Inspiration

As music producers, we feel that music production is a severely underrepresented niche in the field of machine learning and generative AI. We wanted to help supercharge the human race by bringing the newest global revolution to your DAW.

What it does

Melodify is a plugin for the music production software Ableton Live. We utilize the Max for Live platform along with the Ableton LiveAPI to integrate seamlessly with your workflow and provide fresh melodic and harmonic ideas based on the context of the project. Melodify is made possible by streaming MIDI JSON data over UDP to and from an open source GenAI model hosted on Hugging Face. You can try it out in the browser at https://huggingface.co/spaces/skytnt/midi-composer. This is the heart of the entire platform, and after much fine tuning, works as the perfect digital musician's assistant.

How we built it

We used Ableton's "Max for Live" plugin framework along with a python back-end and an open source MIDI data AI model from HuggingFace to create the melody in MIDI.

Challenges we ran into

Parsing json to MIDI: MIDI data is unique in that it has meta messages that communicate more than just the obvious parameters you'd expect from musical data (beyond notes, length of note, velocity), which are encoded in the relationships between the different default parameters. When we were parsing to and from MIDI in order to get an accurate suggestion from the model, it was quite difficult to preserve the complex structure when the MIDI data would have multiple layers of tracks or great length.

Max for Live: The Ableton Max for Live platform has poor support for HTTP/TCP requests, so we had to adapt UDP for a more request/response style API rather than the typical live data streams that UDP is meant for.

Incorporating sponsor tech-stacks into our idea: We tried and failed to incorporate a few technologies into our vision for the project, noticeably one being Fetch.ai. It felt like we hyper-focused on using agents to do the simple back-end and calls to the AI model, so when it wasn't working we ended up having wasted a ton of time.

Accomplishments that we're proud of

A few of us are new to hacking, and we all contributed to multiple parts of the project in different ways. There was a ton of stuff that was new for us. Working with MIDI data. Building an audio plugin. Using UDP. Even though we don't have everything fully integrated I think the parts that we did accomplish we are proud of, especially as a proof of concept.

What we learned

Python back-end dependencies are rough. We learned a lot about how to prevent breaking python dependency trees. We experimented with different solutions like poetry and virtualenv, and used both in this project. We found that poetry had poor support for ML libraries like PyTorch/Tensorflow, especially when it came to edge cases.

What's next for Melodify

Maybe one day we will create a plug in with fully integrated features. We know for a fact that LiveAPI supports insertion of MIDI. So we could make it more seamless and interactive for the user and provide more features.