Virtual Xylophone with Gesture Control over the cloud

Inspiration

We were inspired by the need to play any instrument over the internet with good audio quality while preserving the real time experience.

What it does

It uses touch sensors to detected the notes and sends the data of which note has been touched. This data is being used to reproduce the sound on a remote speaker using python script and cloud. Also, a gesture control system is integrated which gives control of whether the song is to be recorded, when to stop recording and when it has to be replayed.

How we built it

We used three photon boards ,8 touch sensors and a leap motion sensor.

Virtual Xylophone system: This employs a photon board and it detects the note touches and sends this data to the end user. This data is transferred over the internet which enables any number of end users to listen to the music. Any number of users can listen by subscribing to the channel

Gesture Control System: This employs a leap motion sensor and a photon board. The leap motion data is parsed using a python script and is posted onto a serial terminal for the photon board to subscribe to. The gestures inputs help to start recording, stop recording or playback the song that has been recorded. The photon board is used to transfer the gesture command over the cloud

User Speaker system: This employs a photon board and runs a Python script. The photon board receives the cloud data and produces the sound of the notes corresponding to the data received from the virtual xylophone system. The Python scripts helps in producing the sound of the notes and also to record or replay the music based on the commands received from the gesture control system

All the individual systems communicate over the cloud

Challenges we ran into

Making the gesture control and leap motion work. Making Leap motion compatible with Python. Parsing serial data dumped by the python script to be used by photon cloud