AirJam

Music creation for everyone!

Comment

Air Jam

A Hack created at LAHacks 2019, AirJam enables accessible, effortless and fun music creation for everybody.

Inspiration

We originally tried to recreate an Air Guitar hack, which the user can play guitar notes in mid-air without a real guitar. We planned to use computer vision and machine learning to recognize the user's gestures and hand movements, matching the position on a real guitar. Then we realized this has a lot of potentials not only for air guitar performances, but also for people who are interested in practicing guitars but cannot afford one. After creating a guitar experience, we expanded our features to drum kit and xylophone, an automatic interface, a remote visualization, and a dance experience.

Our features

The video above demonstrates our features.

Instrument experiences

Stand in front of a computer camera. Imagine you are playing one of the three instruments — guitar, drum kit, and xylophone, in mid-air. Move your hands and arms just like if you are playing a real one in front of you. Our pose-detection algorithm predicts how you are playing the instrument, and generates the appropriate sounds. You can also see a visualization in front of you, showing which key or drum is hit, helping you learn the instruments.

Smart interface

There is no button or touch interface for our hack. Just stand in front of the computer and play whichever instrument you want. Our machine learning algorithm will recognize which instrument you would like to play.

Remote visualization

Want to share your music with others? We have a web interface that anyone can access remotely and shows the instrument and music you are playing.

Dance experience

Move your body in front of the camera. Your body is the instrument this time! Every move you make creates an 8-bit sound and you are creating your own symphony.

How we built it?

Body movement tracking

We used OpenPose with Python, a machine learning framework developed by Carnegie Mellon University Perceptual Computing Lab, to create a skeleton of the user. By tracking the relative position of various body parts, we can infer the note/beat that the user is playing on the imagery musical instrument.

Experience recognition

At the start of the sessions, a picture of the user is taken and uploaded to Google cloud storage. That triggers a cloud function written in Python, which feeds the picture to an AutoML classification. The algorithm is trained by samples, taken at the hackathon. It classifies which instrument the user is likely to play and start the corresponding experience.

Sound and visualization

Audio files of different instrument notes/beats are triggered to play, when the algorithm detects the user hitting the imaginary instrument. The computer running the algorithm sends an OSC message over a hot spot to another computer. On the other computer, a Processing-based visualization is shown and updates the graphics based on incoming signals to show the imagery instrument playing.

Remote visualization

We have a website with animated content driven by JavaScript. In the backend, an OSC-bridge is running to post the incoming OSC messages from the machine learning algorithm to the front end, thus driving the animation in real time.

Challenges we ran into

  • Optimizing the OpenPose algorithm to run with less delay, achieving the real-time experience
  • Google Cloud Storage + Cloud Function + AutoML takes 10 seconds, could not make it real time to switch instrument experience during a session
  • Machine learning algorithm gets distracted by people walking in the background

What's next

Group experience

We wish to create a group band-like experience, where multiple people can stand in front of the camera and uses different gestures to play all kinds of instruments. Using the experience recognition feature, they can also switch instruments mid-way.

A physical booth

Right now the project is running on two computers. We wish to place them in a booth and locate the booth near tourist attractions or along the beaches at Santa Monica.

Built With

Share this project:

Updates