Inspiration
Was inspired by seeing similar projects, but added a bit more of my own spin on it
What it does
Using a camera, your left hand plays different synth chords based on how many fingers you hold up and sweeps a filter as you move it up and down. Your right hand plays notes and harmonies.
How we built it
We built it in Python and C++ using OpenCV and Google MediaPipe for hand tracking, along with a custom synth engine written in NumPy and sounddevice so it doesn't need an external DAW. We connected it to the Arduino Uno Q over usb-c.
Challenges we ran into
I realized I didn't have a USB C hub to plug a webcam into the Arduino, so we had to route the MacBook's camera and speakers over the single usb-c cable instead. We also had to update our tracking code for the newer MediaPipe Tasks API and smooth out the synth math so changing chords wouldn't make clicking sounds.
Accomplishments that we're proud of
Really glad it works and I was able to experiment with google Antigravity and Opencv
What we learned
Learned how real time audio synthesis and filters work in Python, how to track finger gestures reliably with MediaPipe, and how to communicate with the Arduino Uno Q's Linux processor and microcontroller.
What's next for Virtual Synth
I would like to maybe add a looper to allow further music creation
