PLAI | Devpost

Inspiration

There are many songs out there that do not have dedicated sheet music, and we wanted to create a tool that could let us play any song on the Internet on the piano.

What it does

We are using AI to transcribe music. It converts audio files in mp3, wav, ogg, flac, or m4a format into sheet music by using Google's Gemini 2.5 Pro AI model.

How we built it

PLAI was built using with TypeScript, but we also integrated Google's Gemini 2.5 Pro AI model for audio analysis. We implemented Web Audio API for file processing, and used VexFlow to convert AI-detected notes into sheet music notation. The development involved establishing the foundation, implementing audio processing with drag-and-drop functionality, integrating AI capabilities with error handling, building sheet music generation, and polishing the user experience with custom branding.

Challenges we ran into

It was difficult getting the notes to look like the sheet music we saw online, and it was even more difficult getting the notes to be actually accurate to the ones being played on the mp3. We had a hard time getting the Gemini API model that we wanted.

Accomplishments that we're proud of

We managed to figure out which API works the best for generating sheet music that is easy to read and is also accurate to the song that was put in.

What we learned

We learned that there are a lot of challenges when it comes to analyzing audio files, as not many API's are trained on how to do more complex analyzing tasks yet. We learned how to transform complex audio analysis into an intuitive music transcription tool.

What's next for PLAI

We are planning on expanding what instruments PLAI covers and the length of the sheet music. We would ideally like to expand from snippets of music to covering the entire song, and we would like to improve the complexity of songs that it can analyze. We want PLAI to be able to do a variety of different time signatures, lengths of notes, and clefs. We also want to implement Essentia, which is an open-source C++ library for audio analysis, to make our audio processing even more accurate.