Tonality

Changing the way musicians practice

Comment

Inspiration

After stopping music classes for many years we felt like our playing wasn't improving as we weren't 100% sure what we were messing up on. We wanted to make something that would be beneficial to all musicians whether they be beginners or intermediate players. It would be they're personal music teacher that would provide feedback on their playing.

What it does

Tonality takes in the sheet music file and audio recording of you playing that sheet music and recognizes where you messed up providing the errors that you have made in your playing and a final score of your playing (sees how close you were to playing the music perfectly).

How we built it

For the backend we built it using Python. There were two main things we needed to detect from teh audio recording to compare it to the sheet music, the duration of each of the notes and the pitch. To recognize pitch we tried to detect the note onsets using aubio and then finding the duration based on the difference between consecutive note onsets or the difference between a note onset and a rest onset that are right after each other. For the pitch detection we used a pre-trained model from TensorflowHub. This would help us recognize the musical data from the audio recording, but we still need to access the info from the sheet music. For that we used the library music21 to access each note information converting the sheet music into a dictionary format letting us know the notes and rests in each bar. Then we would compare these two dictionaries formed and see what are the differences based on that we would the find the errors played. For the front-end we used flask and then integrated the back-end with it.

Challenges we ran into

In terms of the back-end challenges we ran into was being able to correctly recognize note onsets from a raw audio file that potentially could have a little bit of background noise. We decided thaat we would only detect note onsets above a threshold that would relative to the entire audio recording's amplitude. Also for the front end we wanted to display a more visual way of the errors. It would be the oirginal sheet music with annotated notes in a different colour representing what you played instead and how that was a mistake, but we were not able to convert musicxml to a pdf or png and there was no way to annotate it the sheet music so that we could highlight certain notes (we wouldn't be able to know the exact coordinates that the note is located in if we were able to convert a musicxml to a pdf)

Accomplishments that we're proud of

Being able to deploy a working back-end and integrating it with the front-end as this our first time making a website.

What we learned

We learned how to use all these libraries and do audio processing.

What's next for Tonality

To be able to work with music that has chords, to add a record feature in the program, and to recognize dynamics as well in the audio recording and compare that to what was provided in the sheet music for each of the notes.

Built With

Share this project:

Updates