After having countless struggles of not being able to find sheet music for songs that we love, we decided to create our own solution.
What it does
The user can input an mp3 file and receive musical notes that correspond to the respective timestamp. After the mp3 is submitted, a snippet of the song is analyzed to determine the artist and song title using the ACRCloud API for Audio Recognition.
How I built it
We converted the .mp3 files into .wav files in order to produce graphs of the Short Time Fourier Transform of the signal using MATLAB's spectrogram function. MATLAB's audio signal processing toolbox was helpful in analyzing and finding the fundamental frequencies of the audio signals.
We wrote two functions, one to convert the frequencies found by the pitch function into musical notes, and the other to find which octave each note was in, in relation to middle C (C4) (262Hz). We outputted three vectors: one for the notes being played, one for the octave of said notes, and one for the time that each note took place.
We ran the MATLAB code from python from a local flask server and then displayed the notes and octaves on the website based on the timestamps.
Challenges I ran into
Some challenges that we had in our program was learning how to successfully use signal processing in 36 hours, however MATLAB's toolboxes helped with that quite a bit. Another challenge was understanding Fourier Transforms and how that related to the readings of our spectrograms and analyzing our audio.
Another big challenge we ran into was live recording the audio and analyzing the audio. We were able to live record the audio but we ran into many difficulties with analyzing that audio.
Accomplishments that I'm proud of
Overall, we are very proud of being able to understand how Fourier Transforms unravel the combination of sound waves back into individual sound waves/frequencies. Along with that, we are very proud of being able to identify the different frequencies on our spectrograms including the fundamental frequency and the harmonics. We were able to understand basic signal processing with no prior knowledge, which is also a great achievement.
What I learned
We learned quite a bit over this project. We learned a great deal about sound waves and their composition, especially how Fourier Transforms can unravel multiple waves back into their original components. We learned how to read spectrograms and how to identify the fundamental frequency along with the harmonic frequencies. We also learned a great deal about basic signal processing, like converting .mp3 files to .wave files and how to perform in depth analysis of the audio.
We learned how to implement the ACRCloud API, requests in backend, and we also became more familiar with the frontend.
What's next for Musical Note Recognition
The next step for Musical Note Recognition is to clean and remove the noise from the audio files to get more accurate results by pre-processing the signals. Getting more accurate results with our frequencies will help us further focus on our analysis; we would definitely utilize some of the functions from MATLAB's audio toolbox to help with this. After that, we will move on from identifying individual notes in our analysis to identifying full chords (strings of notes played together).
The next steps also include being able to successfully analyze a live recorded audio and hopefully also implement AR by showing the chords associated with desired song on a guitar or ukulele. We are currently able to take a live recording and convert a base64 string and write it into an mp3. We had difficulties with formatting issues, but hopefully with more time and effort, we will be able to resolve these issues.
This whole experience was overall very rewarding and we learned a lot at this year's VandyHacks!