Inspiration
WE LOVE MUSIC. WE LOVE MAKING THINGS.
What it does
IT TAKE VIDEOS, EXTRACTS THE AUDIO, TRANSFORMS THE AUDIO FROM A TIME DOMAIN TO A FREQUENCY DOMAIN, TURNS THE FREQUENCY DOMAIN INTO AN IMAGE, THEN FINALLY TRAINED A CLARIFAI MODEL BY PASSING IT THE IMAGES OF THE MAPPED FREQUENCIES IN ORDER TO IDENTIFY INSTRUMENTS.
How we built it
We wrote a java program to scrape sites for URLs of music, sorted by instrument. Using Python, we extracted audio files from the videos and performed Fast Fourier Transforms on them. The FFT took 4 second intervals and outputted the harmonic frequencies in each 4 second interval. We normalized these harmonic frequencies by setting the highest amplitude frequency wave in each interval as 1, and all other amplitudes relative to it. We then converted these frequency files to images. We did this by setting each pixel in the image to a specific frequency and the intensity of the pixel corresponded to the amplitude. Next we whisked the images off to Clarifai for training. By tagging the the images by instrument, we train Clarifai to identify instruments by frequency. This works due to the different overtones produced by varying instruments.
Challenges we ran into
Fourier Transforms are hard. It was very difficult to code for and even after figuring it out, we ran into a wall of errors. We discovered that we actually did not have the computing resources to run the algorithm so we had to use someone's home desktop from our laptop. Additionally, we attempted to use the Clarifai java API--since most of our group knew java and not Python--without any machine learning experience. Although we finally figured it out, it turned out to be too complex and we used the Python API instead. Lastly, we found it difficult to find libraries for Html parsing, scarping, and Fourier Transforms. The end result was that we had to test out many libraries.
Accomplishments that we're proud of
NOT SLEEPING, getting better at table tennis and opening doors, actually getting a working project--which INCLUDES A FAST FOURIER TRANSFORM--which feels really good.
What we learned
Hackathons are fun, TJ can do Fourier Transforms, the correct pronounciation of Fourier
What's next for PITCH
PITCH currently only identifies 5 instruments. The next step would be increasing the number of instruments the model can predict. The ultimate goal would be to include all instruments that are played in the modern day. This would entail gathering more data on instruments to provide to our Clarifai model. Data can be collected on new instruments to allow the model to identify new instruments but can also be collected for instruments that can already be identified. For each bit of data collected, we increase the accuracy of our model. Data for instruments would be affirmative data, telling the model what an instrument looks like as a mapped frequency image, but data for other instruments would tell the model what the instrument isn't, also increasing accuracy.
Log in or sign up for Devpost to join the conversation.