Drum-sets are expensive music instruments. Even if you buy one, it's difficult to practice it at all and any hours because of the noise level. However, there is no denying that drums are extremely cool!
Hence, we developed Humdrum - a virtual drum-set that allows you to practice without buying an actual one and can even be used with headphones. Humdrum uses the computer's camera to detect 17 key points along your body. It then sets up a virtual drum-set around the detected key-points and tracks their motion to simulate the sounds of a Ride, a Snare, a Kick and a Hi-Hat.
In order to detect and track these key-points in real-time, we use a trained PoseNet network built on TensorFlow. In order to simulate accurate drumming sounds and pitches, we incorporate magenta.js.
The biggest challenge we faced while developing Humdrum was mapping 3D drum co-ordinates to a virtual 2D space. It was essential for this mapping to be smooth and accurate for high precision and high recall. In addition, we also faced lag issues while detecting and tracking key-points continuously in a video stream. In the end, we were able to get past these issues and we are extremely proud of our final product. We think Humdrum combines art and technology effectively to instill a passion to learn and practice drumming. In the process, we were able to explore a myriad of libraries that allow integration of the sounds from multiple instruments to create music from scratch using just your laptop.
The next step for Humdrum is availability of drumming tutorials as well as extending the application to virtual reality (VR) users. Right now, it is an application to practice drumming but we hope that in the future, it will be a place where beginners can start their journey into the beautiful world of drumming!