What began as a simple idea turned into the foundations of this project. Learning how to play bass required various tab sheets and music sheets in order to practice. We couldn't find the right sheets of our favorite songs to play so we decided to make some software that would do it for us.
What it does
Synesthesia works by taking an audio file and utilizing an open-source audio separation library called Spleeter to generate a .wav file of the bass components found in the audio file. This .wav file is then used to transpose the waveform into tab sheets
How we built it
A seemingly simple Flask app turned into a monster. Just the simple act of saving a file turned into a painstaking long endeavor only to be stuck among piles and piles of research papers because apparently, this is some kind of novel concept in the field of computing when it actually came to automatic transcription and note detection. Every transformation of information left only more work to do. After what seemed like eons we began the process of placing notes on paper, figuring out the relative math as we went along, an arrangement was born.
Challenges we ran into
One of the major challenges that we ran into was that there was a substantial lack of definitive research in this area. There were a variety of research papers on the topic, however, they mostly covered theoretical approaches to this field.
Accomplishments that we're proud of
The amount of research and time that went into the production of this project is something we are immensely proud of. From what we have seen, this is a one-of-a-kind piece of software.
What we learned
We learned how theoretical most things involved in audio processing really are, and our graphic designer seemingly relearned Adobe Illustrator after 4 years of continuous use.
A unique mask has to be generated which is then used to isolate the desired instrument.
What's next for Synesthesia
Create and utilize our own machine learning model and improve on the accuracy of the transcription.