Detection of body movement using OpenPose
Guitar visualization, blue blocks showing the finger positions.
Drum kit visualization, the circles enlarge when hit
Recognizes which instrument experience to run with Google Cloud AutoML (this shows a sample being fed into the classifier)
Guitar web simulator
Drum kit web simulator
A Hack created at LAHacks 2019, AirJam enables accessible, effortless and fun music creation for everybody.
We originally tried to recreate an Air Guitar hack, which the user can play guitar notes in mid-air without a real guitar. We planned to use computer vision and machine learning to recognize the user's gestures and hand movements, matching the position on a real guitar. Then we realized this has a lot of potentials not only for air guitar performances, but also for people who are interested in practicing guitars but cannot afford one. After creating a guitar experience, we expanded our features to drum kit and xylophone, an automatic interface, a remote visualization, and a dance experience.
The video above demonstrates our features.
Stand in front of a computer camera. Imagine you are playing one of the three instruments — guitar, drum kit, and xylophone, in mid-air. Move your hands and arms just like if you are playing a real one in front of you. Our pose-detection algorithm predicts how you are playing the instrument, and generates the appropriate sounds. You can also see a visualization in front of you, showing which key or drum is hit, helping you learn the instruments.
There is no button or touch interface for our hack. Just stand in front of the computer and play whichever instrument you want. Our machine learning algorithm will recognize which instrument you would like to play.
Want to share your music with others? We have a web interface that anyone can access remotely and shows the instrument and music you are playing.
Move your body in front of the camera. Your body is the instrument this time! Every move you make creates an 8-bit sound and you are creating your own symphony.
How we built it?
Body movement tracking
We used OpenPose with Python, a machine learning framework developed by Carnegie Mellon University Perceptual Computing Lab, to create a skeleton of the user. By tracking the relative position of various body parts, we can infer the note/beat that the user is playing on the imagery musical instrument.
At the start of the sessions, a picture of the user is taken and uploaded to Google cloud storage. That triggers a cloud function written in Python, which feeds the picture to an AutoML classification. The algorithm is trained by samples, taken at the hackathon. It classifies which instrument the user is likely to play and start the corresponding experience.
Sound and visualization
Audio files of different instrument notes/beats are triggered to play, when the algorithm detects the user hitting the imaginary instrument. The computer running the algorithm sends an OSC message over a hot spot to another computer. On the other computer, a Processing-based visualization is shown and updates the graphics based on incoming signals to show the imagery instrument playing.
Challenges we ran into
- Optimizing the OpenPose algorithm to run with less delay, achieving the real-time experience
- Google Cloud Storage + Cloud Function + AutoML takes 10 seconds, could not make it real time to switch instrument experience during a session
- Machine learning algorithm gets distracted by people walking in the background
We wish to create a group band-like experience, where multiple people can stand in front of the camera and uses different gestures to play all kinds of instruments. Using the experience recognition feature, they can also switch instruments mid-way.
A physical booth
Right now the project is running on two computers. We wish to place them in a booth and locate the booth near tourist attractions or along the beaches at Santa Monica.