GestureLabs

Inspiration

Open any audio plugin as a beginner and you see a wall of knobs: wet, decay, feedback, Q, cutoff. Nobody tells you what they do, so you turn them at random until something sounds okay. Producers spend hours on this, and beginners often give up.

We asked a simple question: what if you could feel an effect with your hands, and just say what you want ("make it dreamier") instead of guessing which knob to turn?

That became Gesture Rack: a browser studio where you pinch your fingers in front of a webcam to shape sound, an AI turns your words into effect settings, and short lessons build the intuition behind each effect.

What it does

• Pinch to shape sound. A webcam tracks your hand in the browser. Pinch your thumb and index finger, then open them, and reverb and delay move together in real time.
• Say it, map it. Type or speak an intent like "make it dreamier". Gemini turns it into a mapping (which gesture controls which parameters, and over what range). You see the proposal, approve it, and your pinch now drives it.
• Learn by doing. Bite-sized tutorials (/learn) load a small rack with only the effects being taught, plus step pop-ups like "pinch to raise feedback" or "ask why dub echoes get darker". The full studio stays open for free play.
• Voice control. An ElevenLabs conversational agent lets you request and apply mappings hands-free. [confirm shipped]
• Memory. Backboard remembers your taste and past mappings across sessions. [confirm shipped]
• Session dashboard. Every parameter movement streams into Tiger Data as a time series, and /dashboard charts your session from continuous aggregates.

How we built it

Frontend and audio. Next.js, TypeScript (strict) and Tailwind, deployed on Vercel. Audio runs in Tone.js: Reverb → FeedbackDelay → Filter → Limiter. The limiter can't be turned off, every parameter is clamped to a safe range, and every change uses rampTo so there are no clicks.

Hand tracking. MediaPipe Hand Landmarker runs fully in the browser, so video frames never leave the device. We measure the distance between the thumb tip and index tip and divide it by hand size, so the value stays the same whether you sit close to or far from the camera:

The mapping contract. Everything is built on one small JSON shape: a gesture, plus a list of targets, each with a plugin, parameter, min, max and curve (linear, exp or log). The mapping engine scales $g$ through the curve into each target's range, then clamps to the parameter's hard safe range. We locked this contract in the first hour so four people could build in parallel.

AI. A Next.js route handler sends the user's intent and the rack's parameter schema to Gemini with structured JSON output. We treat the response as untrusted: it is validated with Zod and clamped before anything reaches the audio. Bad output is rejected, never applied.

Data. Tiger Data (TimescaleDB on Postgres) stores param_events as a hypertable. The browser batches events about once per second, and each batch has an id stored in the same transaction, so retried uploads are never counted twice. The dashboard reads a per-minute continuous aggregate, not the raw events.

Privacy. Video and raw audio are never stored. Parameter values and gesture usage are always collected; intent text and effect descriptions are collected while sharing is on. Sharing is on by default, and the toggle sits on the dashboard next to a plain explanation of what is shared.

Challenges we ran into

• Row-level security vs. continuous aggregates. We wanted RLS on every table, but TimescaleDB can't build a continuous aggregate on a hypertable that has row security. We redesigned isolation for param_events: the app role can only insert, a trigger rejects rows whose owner doesn't match the request, and reads go through a function that only returns the caller's data.
• Making a pinch feel good. Raw landmark distance is noisy and changes with distance from the camera. Normalizing by hand size, per-user calibration, and the One Euro filter made it feel smooth instead of shaky.
• Audio that can't hurt anyone. Delay feedback can run away, and reverb decay rebuilds its impulse response (clicks and CPU spikes). We capped feedback below 0.9, throttled decay changes, and kept the live pinch on wet mix and feedback.
• Never trusting the model. ElevenLabs Often would change the diction speech of its output say from a soft voice to yelling

Accomplishments that we're proud of

• A judge can walk up and hear a pinch change the sound in under 30 seconds.
• The app still works with every AI service down: pinch plus rack, with a fake pinch input as a fallback.
• Seeing the smiles on people face when they see the real time effect of them pinching their fingers to the audio

What we learned

• How audio effects really behave: why feedback needs a ceiling, why decay is expensive, and why a ramp beats an instant change.
• Real-time hand tracking in the browser, and how much smoothing and calibration matter for "feel".
• Time-series design in Tiger Data: hypertables, continuous aggregates, and their limits with row-level security.
• Treating LLM output like user input: validate first, then clamp.
• Working as a team of four in 21 hours: lock a contract early, one owner per area, small PRs.

What's next for Gesture Rack

• A Web MIDI bridge so gestures can control knobs in a real DAW.
• Two-handed control and hand height as a second axis. [move to "What it does" if shipped]
• More tutorials, and using opted-in semantic notes to learn what effect settings mean to producers.

Built With

• ai
• database
• machine-learning
• media-pipe
• next-js