This product simulates a soundboard, which is what DJs use to play sounds. It is a square/rectangle with rectangular buttons where each button plays a noise, such as a snare drum or a synth key. They look like this:
These are rather expensive, so we made a cheaper version of this that has infinite customizability. Build A Board allows the user to just grab a piece of paper, draw a soundboard (allows for lots of variance in drawings), and let a webcam watch it, and when it sees that you pressed a button, it emits a noise, just like a real soundboard.
All you have to do is draw a soundboard on a paper or whiteboard with a marker, and aim it at the webcam, like so:
This takes about 20 seconds to draw. Then, to account for lighting differences and small irregularities, we did some transformations to the image so what the computer sees is this:
As you can see, this gets rid of small streaks and localized impurities while keeping only the main parts. Then, the line detection algorithm takes place. It is an aggregation algorithm that starts at the top left corner and follows lines that it finds, including the edge of the screen. Leaving a lot of details out, it counts the contours and registers shapes that way. This is what the computer registers:
The search line flows to the main box and finds it. It then finds the bounding box of the main box and repeats the process for the button boxes, like so:
The smaller boxes are the search spaces based off of where the main box was. Repeating the same process from step 1 for the smaller boxes, we demonstrate a scalable and repeatable solution to finding all the buttons. This is the end result:
From these, the coordinates of the buttons are found. Once the coordinates are found, the program watches for movement in the button spaces and plays pre mapped noises.
All the music is stored into a multi-regional cloud database built with Google Firebase API which stores the .mp3 files the user has played. All the files are connected in real time with a backend Flask server which the user could view in an unique Build-A-Board website. In the website, the user could not only play the music they created, but also share it with their friends or create an entire playlist! This is what the interface looks like:
A lot of technical details are left out, but this is the gist. From this, you can map the 10 buttons to noises. The noises aren’t only limited to ten - for example, we used the top left button to switch instruments so that we could fit 17 noises instead of standard 10. The uses are endless, so go wild.
Challenges I ran into
Because we are looking for the contrast in the image the webcam is capturing, a consistent lighting environment is vital for our edge detection to work accurately. When using our software, aim to find a location with soft light, which would cast diffusing shadows with soft edges.
Computer Vision: Python, OpenCV, NumPy
What's next for Build a Board
We would love to see Build a Board being used to help out the music community. We will continue to build more customization options for existing keys and a larger music library of sound samples.