MaestroSynth

What inspired us?

We're inspired by live performance and casual jamming. We wanted the ability for a live performer to adjust their sound without moving from center-stage, and for someone with a guitar on their couch to change their backing chords seamlessly while focused on practicing their instrument. We were also excited by the opportunity to incorporate MediaPipe's hand tracking API to create something truly new—a motion-controlled synthesizer that's never really been built like this before. Our project aims to serve not only professional musicians but also beginners, functioning as an immersive educational tool for music theory by demonstrating the versatility of the Circle of Fifths in real-time.

Build Details

MaestroSynth uses MediaPipe hand tracking to convert hand gestures and positions into musical notes. With 84 possible chord-note combinations, users can create music simply by moving their hand. The system tracks the user's hand position in polar coordinates relative to the screen center, mapping angle directly to notes on the Circle of Fifths. Additional finger gestures control chord quality (major, minor, 7th, etc.) and a thumbs-up gesture locks the current note for looping, with thumbs-down to unlock. We used MediaPipe's hand landmarking to define functions for detecting finger extensions, thumb position, and fist gestures. The user interface features a Circle of Fifths ring built with Tkinter, divided into four colored quadrants that light up as the user moves their hand. We created quadrant mappings so that hand position relative to center determines which note is selected. Our data pipeline works as follows: hand positional data → note selection → MIDI mapping → rt-midi sends data to DAW → Vital synthesizer receives MIDI and produces sound. A key innovation was switching from cartesian to polar coordinates for hand tracking, allowing us to map hand angle directly to notes on the circular interface.

Our challanges

The lock/unlock function (thumbs up/down) took time to stabilize and debounce properly Getting sound routed through MIDI to our DAW required troubleshooting multiple connection issues Our original gesture logic (thumb direction + hand movement) was too complex—we pivoted to a simpler angle-based approach Merge conflicts during team collaboration required careful coordination Gesture recognition inconsistencies (fist detection, thumb orientation) needed fine-tuning Initially used an outdated backend library and had to restart significant work

What we learned

We're proud of our ability to recognize when our initial approach wasn't working and pivot quickly. Early on, we completely redesigned our note selection system, moving from a complex two-step gesture method to elegant polar coordinate tracking. Whenever we hit roadblocks, we problem-solved by focusing on how the product should actually be used and who it's for—keeping the user experience at the center of every decision. We also learned basic music theory and Circle of Fifths navigation, How to use Cursor as a collaborative development environment, gained experience with hand tracking and computer vision, git collaboration and managing merge conflicts, MIDI communication protocols and DAW integration, and the importance of debouncing and state stability in gesture recognition

Future Improvements

Moving forward, we want to add two-hand tracking. The current product senses one hand, but the potential to recognize both hands opens new possibilities: the right hand could continue controlling note selection on the Circle of Fifths, while the left hand could control harp-style functions inspired by the Omnichord—selecting chord voicings, inversions, or arpeggiation patterns. This would create a truly expressive, two-handed instrument