Inspiration
I'm a musician and sometimes I don't play with sheet music, or I listen to something that I want to play, but can't find accurate scores online. Our with the project was to solve this problem.
What it does
Mus2Tab is a super simple way for a user to take an audio file and transpose it into sheet music specifically tablature for guitar.
How we built it
The first step, we separated the audio tracks to isolate the guitar piece from the rest of the audio file using Demucs an audio seperation model. Then we converted the guitar sounds to midi from pitch and frequency of the isolated track with CREPE, a model for tracking the fundamental pitch of a waveform. Lastly we converted the midi notes into a tablature and a pdf which was all wrapped in fastAPI and deployed on Google Cloud Run with Docker. All of this information was then displayed on a react website for user simplicity.
Challenges we ran into
The most troublesome part of our project was definitely our second part, where we struggled a lot of harmonic frequencies, frequencies that aren't the fundamental frequency (note that you actually played), getting detected on a lot of the attempts we took. We first tried basic_pitch from spotify and had to clean a lot of noise from the audio with no success in removing the harmonics from the results. We eventually dropped basic_pitch and used CREPE along with the noise cleaning techniques we tried previously, which produced passable results.
Accomplishments that we're proud of
Overall, it was an accomplishment to finish our project in a short time period, but specifically we felt that converting the isolated audio file to midi notes was challenging yet exciting. We were able to overcome the challenges of noisy data and iterate between different methods to eventually arrive on something that works. It was super cool to design a website because we are all not very experienced at website building so getting the features to work was super fun.
What we learned
We learned a lot about cleaning audio data, especially relating to frequency. We learned how even small difference in the tone of a track can completely change the results in frequency tracking. More broadly, we learned the necessity of a simple user experience and iterating throughout the design process.
What's next for Mus2Tab
We believe that Mus2Tab can be scaled by adding compatibility with more instruments, giving a greater number of people access to our tool. Also, being able to handle polyphonic audio (chords) would increase its power from just handling a single melody, along with continuing to reduce noise from the file. One more challenging future plan would be supporting direct input from the user and converting their music to tablature in real time.
Log in or sign up for Devpost to join the conversation.