We're all huge fans of Guitar Hero but we're not musicians. So we created a real-time system that lets you play any instrument and create the music you like! We made the system super intuitive and fun to play with.

What it does

  • Make music using your body.
  • Play different instruments.
  • Can be connected to any virtual instrument (VST) or digital audio workstation (eg: Ableton, Bitwig Studio, ...) through MIDI (music protocol for sounds).

Demo Videos:

How we built it

  • Using the OpenPose library to extract the skeleton of the body of a person.
  • Tracking right and left hands over frames.
  • Using signal processing to detect events (position of the hands, note on, note off)
  • Converting the events into MIDI signals.
  • Connect it to a virtual intstrument Let's play music !

Challenges we ran into

  • Because the skeleton extracted from OpenPose can be noisy, processing the signal and detecting events in a robust manner that works with anyone and in any conditions is quite tricky. We applied signal processing techniques to filter the signal and improve the detection of events.

Accomplishments that we're proud of

  • Putting all the pieces together and having a working prototype.
  • Opening new ways of creating music.

What we learned

  • To use the OpenPose Library
  • To process noisy signals
  • To use the MIDI protocol to control virtual instruments.

What's next for Air Guitar 2 Sound Converter

  • Use a laptop with NVIDIA graphics or a desktop PC and make it run in real time.
  • Improve the event detection and recognition using more complex techniques to make it even more robust.
  • Support multiple players at the same time.

Built With

+ 22 more
Share this project: