Inspiration
Sometimes you'll see videos of professional musicians and wonder how they did it, what kind of bowings did they use, what finger patterns were they playing, and how were they interperting the music. That's where Soli comes in. We want to give people the tools to emulate the very best by showing them sheet music that follows the exact performance.
What it does
Our project analyzes videos of performaces and transcribes them, using video footage for fingerings and bowings, and audio for notes and rhythm, into readable sheet music that can then be used when practicing or learning a new piece.
How we built it
Initially we conducted some research into possible AI models that would be able to process the audio and convert it into notes, which resulted in us using MERT. We then, using Cursor, implemented the MERT model using python as the backend of our website to transcribe the audio. After that, we implemented video recognition to track hand movements in order to monitor fingerings and bowings. Once we had everything up to snuff, we integrated the backend video analysis with a front end we had been developing concurrently.
Challenges we ran into
We had difficulty integrating video and audio analysis to form a coherent output. Additionally, we had trouble getting getting audio analysis to properly identify notes and occasionally it would produce out put that was an octave or a fifth off. Towards the end we experienced scope creep and had to better prioritize what features were central to our project
Accomplishments that we're proud of
Audio transcritption is better than we were expecting and is also capable of identifying well known songs.
What we learned
We learned how to integrate Cursor into our workflow and we learned how MERT works as well as how video and audio analysis is performed.
What's next for Soli
We plan to improve the accuracy of the analysis as it can occasionally misinterpret things.
