Aeria

Inspiration

I love music and I'm learning to play the piano but I don't always have time to sit in front of a real piano or a real instrument and play. So I came up with the idea of Aeria, a browser-based, gesture-controlled music instrument (with many instruments😭😂). I wanted to be creative and do something that nobody else (I think) could have come up with (especially the theremin which sounds like a ghost when I play it by the way👻😂).

What it does

It plays a note of music (from C all the way to B) with hands gestures which are detected through the webcam in real time through Mediapipe and a silhouette of 21 points. Each finger plays a different note and the higher the hand is, the higher-pitched the note becomes. There are many instruments such as piano, pad, violin, bells, theremin , and each of these instruments make a visual feedback (particles, animated keyboard, skeleton). There is also a feature which allows to play with pinched fingers (the thumb and the index together). It needs no hardware, has no cost and runs in the browser.

How we built it

Aeria is built using React and Vite for the frontend, MediaPipe Hand Landmarker for real-time hand tracking (no video data is ever leaving the device), and Tone.js for audio synthesis. I built the entire project with Codex, across multiple focused sessions. The project structure was set up in minutes, allowing me to jump straight to the most interesting part: integrating MediaPipe and implementing live hand-skeleton detection. The real iterative work with Codex focused on the gesture-to-note mapping system; translating finger positions into coherent musical notes is no trivial task. An early attempt using free mapping sounded chaotic, so I decided to constrain the notes to a specific scale, first with pentatonic and later expanding to a full major scale, to ensure that any gesture would sound harmonious, regardless of the player's precision. I also created a dedicated design system (color palette, typography, and signature visual elements like the tempo-synced waveform) and I worked with Codex through multiple debugging sessions to fine-tune gesture detection.

Challenges we ran into

Fine-tuning hand tracking (😭) was so hard, because in the beginning, one imprecise movement made the whole thing sound chaotic, so I needed to adjust the threshold of the detection and hand-presence confidence permanently till it started sounding like a real instrument with the right gestures and the more or less precise movements. Then, there was the low-light latency 😑which was rather hard to handle because that was not a problem in the code but one of Mediapipe's limits.

Accomplishments that we're proud of

I have an instrument player with hands that works on a computer without any API key call, and working through the webcam (yay🎉). There are multiple playing modes, with accords and theremin (👻😂). Annnnd the instruments visuals that are reaaally close to 3D hehehe.

What we learned

I learned the concept of hysteresis as applied to user interaction: using two different thresholds (one to trigger, another to stop) rather than just one, to prevent a slight hand tremor from causing a note to flicker on and off repeatedly. Regarding audio synthesis, I learned that the Web Audio API (via Tone.js) imposes constraints such as the requirement to start the audio context only after a genuine user gesture; otherwise, you get total silence without any explicit error message. Finally, working with Codex on a creative, interactive project taught me to craft prompts in a very different way: instead of describing an abstract feature, I had to describe a physical sensation and iterate through real-world testing rather than by reviewing code.

What's next for Aeria

The next step for Aeria is real-time collaborative mode, enabling multiple people to play together remotely, whether for a choir rehearsal, a melodic brainstorming session, or simply a duet improvisation without being in the same room. I would also like to add automatic key detection based on humming, allowing users to sing the desired mood and have Aeria automatically adjust to it. Finally, a robust export and sharing feature for performances would let users keep a record of their creations and share them easily, beyond the live session itself and I am even considering creating a comprehensive library of musical gestures for instruments, particularly for the piano.

Built With

• codex
• mediapipe
• react
• vite
• webcam