AirDeck

The Landing Page With Interactive 3D Models
The DJ Deck In The Middle of Mixing
DJ Mixer with Track Holding and Overlay Filters
Air Pong, one of Five Arcade Two Player Games
AirMix Gameplay, osu! like Gameplay Based Automatically Off The Tracks
Secret Beta DISCO Overlay

Inspiration

We wanted music and games you play with your whole body, but with nothing in your hands. Just Dance was the starting point: the fun of moving to the beat, here with your hands in front of a webcam instead of a controller. The visual style comes from the Windows XP "Bliss" wallpaper and the Frutiger Aero era (blue skies, soft clouds, glossy glass buttons). That's the world of our landing page. We also took ideas from these hand-controlled music and interaction clips:

What it does

AirDeck turns your webcam into a DJ controller and a game console, right in the browser:

DJ Deck: pinch the air to change speed and filter; the distance between your hands is the volume. Fist gestures switch effects (reverb, echo, bitcrush, flanger) and live visuals.
AI DJ: Tell the AI DJ a vibe ("chill into hype"), and Gemini picks the next songs while ElevenLabs gives the DJ a voice.
Arcade · 2P: two players, one camera, five hand-tracked games: Snake Battle, Bubble Trails, Air Pong, Bubble Pop and Hand Duel (rock-paper-scissors on the beat). -AirMix: Turn any playlist into an osu!-style interactive rhythm game played with your fingertips, where hitting targets live-mixes the tracks, controls filters, and matches the beat.

How we built it

Hand tracking: MediaPipe Hand Landmarker runs in the browser and tracks up to two hands in real time. A One Euro filter smooths the landmarks.
Audio: everything goes through the Web Audio API and an AudioWorklet: effects, tempo and downbeat detection, beat-matched blends and echo-outs between songs, plus an arcade chiptune generated entirely in code.
AI: a small Python server asks Gemini for song picks as structured JSON and turns the DJ intros into speech with ElevenLabs. API keys stay on the server, which only answers localhost.
Visuals: Three.js for the 3D club, WebGL shaders for the hand-framed effects, and Canvas 2D for the arcade.
Music: yt-dlp builds the song library and lets you add any YouTube track.

In the rhythm game, a hit counts as Perfect or Good depending on how close it lands to the beat:

$$ |\Delta t| \le 160\,\text{ms} \Rightarrow \text{Perfect}, \qquad 160 < |\Delta t| \le 320\,\text{ms} \Rightarrow \text{Good} $$

Challenges we ran into

Jittery hands: raw landmarks shake. Without smoothing, the speed and filter wobbled with every tiny movement.
Two players, one camera: the image is mirrored, so we split the players by where each wrist is on screen.
Controls that feel natural: our first Snake turned on a grid and was too hard to steer by hand. We rebuilt Snake and Bubble Trails so they follow your fingertip.
Mixing songs live: beat-matching tracks with different tempos meant detecting BPM and downbeats from the audio itself, and falling back to an echo-out when the tempos are too far apart.
Depending on external APIs at a live event: Gemini can hit rate limits or lose connection, so the DJ falls back to picking songs by genre and tempo, and the browser's voice replaces ElevenLabs.
Teamwork Compartimelization: we built several pages in parallel on separate branches, so keeping shared links, file renames, and one visual identity in sync took constant coordination.

Accomplishments that we're proud of

-Fun and Unique Visuals: were able to pivot away from overused AI frontend visuals and develop a unique style harkening back to early 2000s technology. -DJ Mixer Expansiveness: the DJ air mixer was able to produce multiple outputs on its limited scope of gestures. Gestures were kept simple and quick to pick up, but still included advanced DJing concepts such as visuals, two-track mixing, and mix holding were integrated. -Developed a Robust Audio Engine: allowed for the manipulation of audio files based on a large combination of parameters, and allowed for automatic track mixing based on the original audio output.

What we learned

How much you can do in the browser alone: real-time ML hand tracking, audio DSP and 3D, with no installs.
Designing for the body is very different from designing for a mouse. The gesture has to feel natural first, and the precision comes after.
Always build a fallback around an AI API.
Optimizing machine-learning model weights for even lighter client-side performance on mobile webcams.
A strong visual identity (Frutiger Aero) helps a team project feel like one product.