Paper Drumkit

The Setup
Computer Tracking
User Interface in Browser

Inspiration

I somewhat self-identify as a frustrated songwriter who loves writing lyrics to songs and coming up with melodies, but I don't have the skills nor the training to actually compose instrumentals. I would love to learn to compose during my spare time, but music equipment sure costs a lot when I'm unsure how much use I would be able to get out of it. Therefore, I came up with the idea to come up with a more accessible music tool for myself.

What it does

Paper Drumkit makes paper smart. Using only a standard webcam, a user is able to use a piece of paper to compose beats. This allows a user to learn to compose music cheaply before committing to buying an expensive drum machine (or other heavy-duty music studio equipment). In addition to using paper to physically play the music, the user is able to use a menu in their web browser to upload the sound clips they want to use to each button, as well as preview their uploaded sound clips.

Currently, the paper canvas is set such that there are 6 buttons that each play drum sounds. However, this is extensible as not only can the user upload whatever sound they want to each button, the display and code can also be slightly changed so that the paper canvas set up mirrors a piano or other instrument.

How I built it

I programmed the project primarily in Python (which was fun, since this was my first time using Python). The browser UI is programmed using HTML and CSS. The computer vision framework used was OpenCV.

Challenges I ran into

Having the camera track my fingertip while ignoring noise in the image was a problem that took me a while to solve, as was the problem of how the program, using purely visuals (no distance sensors or anything like that), would be able to sense that the user has tapped the paper. This is still a work in progress, as change in environment lighting as well as time lag makes the tapping a little finnicky.

Accomplishments that I'm proud of

Simply making it work and having it play music is such an accomplishment for me! Having learned Python in the process is a wonderful bonus a well.

What I learned

I learned how to code in Python and how to make a basic web app using Python, served by Flask. I learned more about computer vision and am confident that, in the future, I will be able to move on to more complex camera-based projects, whether they be based on the computer or phone.

What's next for Paper Drumkit

On the functionality end, I need to optimize the time between a click and a note playing -- there is currently a variable time lag between the point that a user's finger touches the paper and the point that a note actually plays. Additionally, at the moment, the program is not able to play notes concurrently; in order to do this, I will need the camera to track at least two fingertips at a time, and I will need to find a different audio plugin that will allow me to play two notes simultaneously. Finally, on the user interface end, the simple web browser interface that I have made for the demo is sufficient for the project at its current stage. However, moving forward, it should offer additional functionality, the next one I have in mind being that a user is able to save the sequence of notes that they played on the paper drumkit.