The theremin is an enigmatic musical instrument, unlike most instruments in that it is able to produce noise without contact. This idea of noise without contact led us to wonder if a physical instrument was even necessary at all. As a team, one of our main interests is in Computer Vision, so we were inspired by the Theremin to use Computer Vision to create a device that could produce music without physical touch. Enter the Handstrument, a program that plays music with just the wave of your hand.
What It Does
Using Computer Vision, the Handstrument tracks a user’s hands to produce music.
How We Built It
The project was built to support both Intel RealSense D400 and Xbox Kinect RGBD cameras. By processing camera frames via openCV, mediapipe, libfreenect, and pyrealsense2 we were able to perform pose-tracking in real time. From pose-tracking information we then mapped joint positions, velocities, and accelerations to certain sounds to produce a “virtual instrument.”
Challenges We Encountered
Our biggest challenge was connecting our front-end with our driver code and displaying the different instruments in a clean user interface. Most of our team members have limited experience with front-end design tools. Most of our time was spent learning how to use these tools and supporting modules.
Another challenge we faced was managing the obscure dependency requirements of our software. Unfortunately, much of the software used to interface with our cameras are outdated, so we relied on software that might not be as accessible.
Accomplishments That We're Proud Of
We are proud of our simple, yet effective method of using skeletonization to retrieve pose-tracking information (position, velocity, acceleration). We are also proud of our models that we believe accurately capture the behavior of Bongos and a Theremin. Our team was driven to produce software that precisely emulated these instruments. We worked tirelessly to balance thresholds and eliminate lag to create a fun and seamless experience for the user. Despite our initial lack of experience, we persevered in developing this amusing product.
What We Learned
We learned how to create a GUI in python that connects a camera feed stemming to our development on openCV. Most of us also are beginners in Python, and through this project, have grown closer to the language.
The Future of Handstruments seeks to add more instruments and features. The possibility of adding more features such as multiple players, for duets, trios, and bands. The team also speculated implementing a conductor feature where your hands would control the sections of a symphony or orchestra. We also want to add more support to artists and creators, by adding a recording feature.