We wanted to combine our interest in music and object detection software.

What it does

The wavechord detects your hand position above the Kinect and maps your position to a corresponding chord on the piano. Currently, the synthesizer generates I, IV, and V major chords for simplicity.

How we built it

We built this using OpenCV, PyAudio, Kinect, and a Raspberry Pi 3 B+.

Challenges we ran into

The Raspberry Pi has limited computational headroom, so we were unable compile the pytorch library to link our convolutional neural network we trained to detect the hand gestures. We used some open source code for hand detection instead.

A pre-trained model in PyTorch would have allowed us to have more control over the notes, pitch, etc.

Accomplishments that we're proud of

  • We worked around a limited timeframe and resources and create a useful chord player.
  • Implemented a CNN to detect sign language for letters A to I using pytorch on individual images.

What we learned

  • Compilation on the Rasberry Pi takes a longgg time. Configuring more swap space is very necessary when using cmake on a system with limited RAM like the Pi.
  • The Kinect is difficult to configure on the Pi. Even with OpenCV compiled with FFMPEG support, we couldn't get a RGB colour channel. Instead, we used the fswebcam CLI program to capture images with the Kinect at around 1 FPS.
  • You can create chords via superposition of independent frequencies. We used PyAudio and NumPy to generate chords that traditionally sound nice together (i.e. I, IV, V)
  • To install PyTorch on the Pi, we would need Python 3.6 or higher. Building Python 3.6.5 failed twice (~4 hours). PyTorch supposedly takes 12 hours! Space was limited as well, so configuring more swap space would have been prohibitive. Next time we could maybe use a VM running raspbian stretch to build Python and PyTorch.

What's next for wavechord

  • An an improved input mechanism.
  • Improved sound engine (Pi audio was difficult to configure. Jack server?)
  • Maybe use a different audio library, such as Pyo (DSP)
  • Depth sensor support with Kinect - i.e. vibrato, pitch bending, volume adjustment, aftertouch simulation
Share this project: