Inspiration
To learn an instrument, most people like to play popular songs they know. But finding good fitting free sheet music turns out to be incredibly difficult because its often paywalled, and solid arrangements for your skill level are few and far between. Say you want to learn a piece of music. If you cant find fitting sheet music, which happens most of the time, there is literally nothing you can do. You just cant play that piece. And at least in our experience that has been incredibly demotivating and inhibitory to learning music.
The goal is to democratize music.
What it does
The user can create an arrangement for any combination of instruments and skill levels for any song. It scrapes sheet music from the internet that's adjacent to the arrangement, then feeds it into our complex conductor agent we imbued with a lot of music theory to create a first pass, synthesizing the input midis into a playable version for the target instruments. That then gets fed into a critic for reinforcement learning that scores it based on quantitative values like rhythm density, which has never been used in this context before, and well grounded qualitative metrics. We repeat that cycle until the improvements plateau and return the sheet music to the user. The user can then select measures and ask the AI to revise it to improve the arrangement.
How we built it
The front end is based on React. We used OSMD to generate Sheet music from XML files and we used firebase to store generated songs so they are not lost once the website is closed. The custom music player uses Tone.js to playback music with Magenta to make it sound like real life instruments. The backend incorporates Gemini for a multitude of things, including checking potential sources for web scraping. From here, we generate a “batch” of sources which will then be scraped by Selenium (python). However, MIDI files generally differ in key, section, instrument tracks, etc. We use further AI screening to filter out for only quality and standardized data. From here, we feed convert MIDI into JSON event data by using a python library to read specific keypresses, building the MIDI from event data. Then, the JSON data is fed in a feedback loop led by a prompt and then an evaluation/punishment model that prioritizes similarity to original MIDI tracks with a classification model that determines consistent melody lines and features across scraped MIDI. Then, the JSON is quantized using some math and python before turning it back into MIDI and using MSCORE CLI to turn it into functional, react-cooperating XML script
Challenges we ran into
The AI agent was so difficult to make and it didn't look like AI had any sense of musical reasoning for a looooong time.
Accomplishments that we're proud of
The AI agent system that we built that can create truly cohesive, playable, and good sounding sheet music . That has never been done before.
What we learned
We developed experience in building AI pipelines, connecting multiple models and agents to handle tasks such as music analysis, generation, and file transformation. This required working with audio file manipulation, including converting between formats like MIDI and MusicXML, to make data interoperable across tools.
What's next for Virtuoso
If AI can generate good-enough arrangements for every instrument/difficulty combo, we can then build out "Wikipedia for sheet music", allowing human arrangers to make each score good, over time building a massive library of free sheet music.
With the novel musical understanding we developed and the ability to manipulate sheet music (to scale up difficulty, etc), we could create a really really good music teacher.
Log in or sign up for Devpost to join the conversation.