Deep Beep

Here is the bounding boxes for the notes
Here is an example of the staff line detection

Inspiration

We wanted to learn more about image processing, by using open cv to analyze sheet music. It seemed like a neat and relatively simple idea. Little did we know, sheet music is full of surprises.

What it does

Takes pictures of sheet music and plays them out loud using MIDI sounds.

How I built it

We broke the problem into three parts. Part one, we find the staffs of the sheet music (the horizontal lines that distinguish each note. We accomplished this by utilizing a common technique in computer vision called the Hough transform. This function takes a point and determines the potential lines through that point. By comparing potential lines from each point, we can find prevalent lines of the image. The next part of the project is note detection by finding the dots of the notes then extrapolating the x and y coordinates we can use the positions of the lines to determine the pitch of each note. Finally we made a Qt based app to play the detected notes.

Challenges we ran into

Dealing with inconsistencies of captured images proved to be a big challenge we only mostly overcame. Our staff finding algorithm is only 80% accurate right now. The eighth notes bars also caused some problems. Detecting the notes also was tricky as our first several methods yielded nothing. We never got to classifying note types (half, whole).