Air Drum Sticks

Inspiration

Going with the theme of the hackathon, we wanted to combine some things from the past and come up with something wonderful. We are also a big fan of 90's rock music and like the drums and guitars. We are also a huge fan of Magic-themed movies like the 1939 The Wizard of Oz and the Harry Potter Series. So we decided to combine the magic wand from the magic movies and our favorite 90's rock music for this project.

What it does

It is a device that can recognize hand gestures and send the recognized gesture signal to the PC using Bluetooth. The PC is running a Python program that will take the recognized signal and based on that, it will play various tunes based on rock music.

How we built it

We built it using an ESP32 board and used an MPU6050 IMU sensor to recognize the signals.

We collected different gesture data from the ESP32 and the IMU sensor.
We used the collected gesture data and trained a TinyML model.
We deployed the trained TinyML model into our ESP32 board for inferencing.
We used Bluetooth communication for sending the data of the detected signal to the PC.
Based on the gesture detected, our Python code running on the PC plays a tune.

*Fun Fact:: The small ESP32 board is actually running a Machine learning model for inferencing to detect the signals from the IMU sensor. *

Challenges we ran into

The greatest challenge we faced was in finding a communication protocol to send the data from the ESP32 board (The Air Drum Sticks) to the PC wireless. We eventually came up with using Bluetooth communication for our project.

Accomplishments that we're proud of

We can play around with the device by waving it around like a magician to form different gestures and also play some cool rock music.

What we learned

We learned a lot by working on this project. Most importantly we got to experiment a lot with working on sending the data from the ESP32 board to our PC. We found Bluetooth communication to be the most reliable and went ahead with it.