Pianotes

Inspiration

I love playing the piano and am often inspired to learn new songs - whether its from a video game, movie, tv show, or whatever I'm listening to at the time. But often I (and many other musicians) run into a problem: WE NEED SHEET MUSIC! This boils down to a number of common issues:

1) We could play by ear, but we can't play by ear that well - so we need sheet music.

2) We learn and can play the song pretty well, but we don't know how to make sheet music ourselves - so we need sheet music.

3) We find the song we love online such as through YouTube and are excited to learn it - but it turns out the other talented individual is also playing by ear, or you have to pay a crazy amount for the sheet music they made! Fair enough, but we don't want to have to pay - so we're still in need of sheet music.

Anybody else relate?

What it does

Pianotes is an (in progress) application that aims to convert an audio file (e.g. .wav, .mp3, .mp4) into readable and editable sheet music. Using Google Magenta's Onsets and Frames Transcription Model, an audio file can be accurately captured and converted into a .midi file. Thereafter, the .midi file is parsed for note events and converted into a .musicxml file, a common open format for digital sheet music that can be uploaded and edited in industry standard applications such as MuseScore and Sibelius.

How we built it

Following Google Magenta's documentation, a script using the Python subprocess was created for the wav_to_midi.py conversion. A separate script was built there after for the midi_to_xml.py conversion. A user will also be prompted to input the tempo of the song during midi_to_xml.py conversion to ensure notes are placed on the sheet music more accurately.

Additional Python scripts in were created to validate the state and tempo of the converted files in tempo_analysis.py and xml_validation.py.

Challenges we ran into

1) Initially the goal was to try and build an ML model from scratch that can convert .wav to .midi, but making one is extremely hard! Using a pre-built model through Google Magenta helped, but the problem then shifted from converting a .midi file and making it more readable on programs such as MuseScore.

2) From zero knowledge, understanding audio engineering and different music file types and what they represent.

3) Sustained pedaling effects - maybe the most challenging part of the entire process. This makes converting a big headache, as many unnecessary ties are introduced in the resulting sheet music (see thumbnail lol) and cause many notes to have the wrong timing or length. Going through this really shifted the goal for the project and it required tradeoffs of making the most accurate conversion from the original audio file, or sacrificing accuracy for better overall sheet music readability.

Accomplishments that we're proud of

1) I was able to at least create sheet music that is somewhat readable and is a good place to start! Even if it's only 50% accurate, that's a least a big help for those looking to make sheet music compared to starting from scratch.

2) Learning a lot about audio engineering and ML models that work with audio files! I read through a lot of papers and though I still don't understand a lot, it was really fun and fed into my music passion.

What's next for Pianotes

1) Building a FE/BE and deploying it! I wanted to do this but I was really invested in the conversion part as it was more unfamiliar territory. Hopefully I can get around to this soon, I'd love for everyone to try it out :)

2) Really cracking down on the .midi to .musicxml conversion. I have a couple ideas on how to clean up the timing and improve the accuracy of the script.

3) Testing with more types and genres of music - I've only been working with a couple of .wav files and I'd want a solution that can convert any style of music relatively well.

Built With

• google-magenta
• python