Inspiration
Rhythmic cueing (walking to a steady beat) is a well-studied tool in gait rehab. Adaptive versions already exist, like MedRhythms InTandem, but they need dedicated shoe sensors. I wanted to see how far you could get with just a laptop webcam and a browser.
What it does
Andante watches you march side-on through your webcam and measures your cadence (steps per minute) and form (trunk lean and knee lift).
- When your steps are steady, it starts a beat at your own pace
- While your form holds and you're keeping up, it nudges the tempo up a few percent at a time
- When your form slips, it eases back down, never below your starting pace
- Every so often it runs a carryover check: the beat goes silent to see if your rhythm holds without it
- Music layers build on top of the beat. They thin out when form slips, and dip when you're off the beat so the kick is easier to follow
- A small on-device ML model learns how you respond: how closely you follow a tempo increase, and when your form is about to slip, so it holds instead of pushing
- After the session, a Gemini coach reads the numbers and writes a short report plus a plan for next time
The design idea: guide the pace, don't chase the feet.
How we built it
- Pose + steps: MediaPipe Pose Landmarker running in the browser. Steps come from the gap between your feet, using adaptive thresholds, with no per-person calibration. Cadence is the median of recent strides.
- Beat + music: Tone.js. The beat runs on its own clock, and your cadence only nudges the tempo, so the beat stays steady even when tracking jitters.
- Tempo control: a rule-based controller handles safety: a floor at your starting pace, a ceiling at +15%, easing off when form slips, and holding when you're off the beat.
- Per-person model: trained live in the browser. It's a weighted estimate of how much of each speed-up you follow, plus a 4-feature logistic regression (SGD) that predicts a form slip a few seconds ahead. It warm-starts from your past sessions and falls back to the rules until it has enough data.
- Gemini coach: after the session ends, a Vercel serverless function sends a session summary to the Gemini API (only numbers, never video). It asks for structured JSON output that follows a fixed schema: summary, what went well, what to work on, trend across sessions, what the model learned, and next-session targets. The prompt keeps next session's targets close to this session's. Gemini never runs during the live session.
- Privacy: video never leaves the browser.
- Stack: Vite + TypeScript, Canvas charts, Vercel.
Challenges we ran into
- Counting steps reliably from a side view at very different speeds. Slow marching kept getting missed until I switched to stride-based cadence and a longer window.
- Measuring form without labels. Lift is scored against your own live range, and lean against a baseline it records right after the beat locks.
- Stopping the controller from fighting the user: speeding up before form was measured, getting stuck at the floor, and false "off beat" readings.
- Honest carryover scoring. Standing still now scores 0% instead of being skipped.
Accomplishments that we're proud of
- Counted 40/40 steps at 80 bpm against a hand count
- Pace held 91–103% in carryover checks in my own testing
- Follow rate 57–81% across sessions
- The whole live loop runs on-device in real time
What we learned
Keeping AI out of the real-time loop made everything simpler and safer. The beat needs to be dumb and steady, and the smart parts should advise, not drive.
What's next for Andante
- Testing with physiotherapists to see whether the form cues match what they look for
- A treadmill and running mode
- A progress view across weeks
- Better tracking in low light
Built With
- gemini
- html5
- javascript
- mediapipe
- tone.js
- typescript
- vercel
- vite
