home
lesson-plan
its-a-murder
tracking
voice-coach
full-screen
lesson-plan-2

Inspiration

Learning choreography from a video means pause, rewind, squint, guess, repeat. Nobody tells you what's wrong on your attempt. Dance games only score their own routines, practice players can loop and slow down but never watch you and teachers can't watch every rep at home. We wanted the routine you care about, from any video, with a coach that actually watches.

What it does

Upload any dance video (up to 10 minutes). Shadow finds the beat, deletes the audio, extracts the dancer's skeleton, and cuts the routine into 8-counts (or 4-counts for short clips). It names the moves and learns repeats once.
It builds a lesson plan with Gemini: learn each 8-count on its own, join them up at checkpoints (1–2, 1–3 … the whole routine) and earn bronze, silver and gold as you reach 0.5×, 0.75× and full speed. Tell the coach "I have two weeks before my show" and it re-plans.
You practice in one continuous flow. Press Start and it goes full screen: the coach counts you in on the beat in its own voice, the original video plays beside your camera and a 3D ghost dancer is laid over you, your limbs turning green, amber or red.
The murder meter. A group of crows is a murder. Every slip sends a crow to a perch above your camera: "Just a crow…", "An attempted murder…" and at three, "It's a murder!". A 2-second cooldown means one slip is one crow. Then Shadow stops, shows exactly what to fix ("Right arm, a touch higher. Match the ghost."), freezes the ghost on the target pose, and carries on the moment you match it.
Hands-free voice coach. Say "slower", "again", "show me count 5" and it happens instantly; ask "how do I fix my arms?" and the coach answers out loud and acts on it.
It tracks you in a crowd. Several people can be in frame; Shadow keeps coaching you, even if someone walks past or you step out and back.
Progress across routines: streaks, steps passed, practice per day, and habits that follow you between routines ("right arm too low on count 3").

How we built it

Browser: React + TypeScript. MediaPipe pose tracking runs locally. Each person gets a fingerprint from an OSNet re-ID model off Hugging Face, run in a Web Worker and combined with clothing colour and position. A lag-tolerant mistake detector drives the crows. The 3D ghost is Three.js.
Backend: FastAPI. The upload pipeline runs ffmpeg, then librosa finds the beats, then the audio is deleted, then pose is extracted and the routine is cut into counts. A learning engine handles mastery ratings, spaced review and k-means habit clustering. We tested scoring against AIST++.
Google Gemini: writes the lesson plan, re-plans with tools, and answers mid-practice questions with validated actions (speed, show pose, redo, skip).
ElevenLabs: the coach's voice, with the count-in timed to the beat, one voice at a time and cached lines.
Tiger Data: every rep in a TimescaleDB hypertable, so streaks, progress and habits are fast time_bucket queries.
Clerk: login, with all data kept per user.

Challenges we ran into

Learners dance a little behind the music, so we built lag-tolerant matching.
Keeping the coach, the count-in and the mic from talking over each other.
Tracking the right person in a busy hall.
Getting counts from clips with no music.
Scope Creep

Accomplishments that we're proud of

It works end to end on real footage: upload, lesson plan, live coaching, then progress. The crows turned out to be the most fun part, and the video stays private.

What we learned

Computer vision: pose models give you keypoints, not judgement. To compare a learner with a routine, we normalised skeletons and compared limb directions rather than joint positions, so it works for any body size.
Tracking: clothing colour alone swaps people and position alone loses them. Adding OSNet re-ID embeddings, a 512-number fingerprint per person, made identities survive crossings and exits. Running the model in a Web Worker kept the camera loop fast.
Timing: people follow a beat behind the music, so matching each frame to a window around "now" mattered more than any model.
AI pipeline design: let the AI decide and the tools measure. Pose, beats, alignment and scoring are deterministic and tested. Gemini plans and explains using their results, returns structured actions we validate, and falls back to rules when it's slow or wrong.
Voice UX: latency and overlap matter more than voice quality. We cached ElevenLabs lines, scheduled the count-in with Web Audio so it lands on the beat and allowed only one voice at a time.
Data: comparing professionals on AIST++ showed that "a good score" is hard to calibrate. The distributions overlapped, so we kept the thresholds hand-tuned and honest.
Coaching: motor-learning research holds up. One specific cue at a time, only when it matters, beats constant feedback.