Inspiration

Rhythmic cueing (walking to a steady beat) is a well-studied tool in gait rehab. Adaptive versions already exist, like MedRhythms InTandem, but they need dedicated shoe sensors. I wanted to see how far you could get with just a laptop webcam and a browser.

What it does

Andante watches you march side-on through your webcam and measures your cadence (steps per minute) and form (trunk lean and knee lift).

  • When your steps are steady, it starts a beat at your own pace
  • While your form holds and you're keeping up, it nudges the tempo up a few percent at a time
  • When your form slips, it eases back down, never below your starting pace
  • Every so often it runs a carryover check: the beat goes silent to see if your rhythm holds without it
  • Music layers build on top of the beat. They thin out when form slips, and dip when you're off the beat so the kick is easier to follow
  • A small on-device ML model learns how you respond: how closely you follow a tempo increase, and when your form is about to slip, so it holds instead of pushing
  • After the session, a Gemini coach reads the numbers and writes a short report plus a plan for next time

The design idea: guide the pace, don't chase the feet.

How we built it

  • Pose + steps: MediaPipe Pose Landmarker running in the browser. Steps come from the gap between your feet, using adaptive thresholds, with no per-person calibration. Cadence is the median of recent strides.
  • Beat + music: Tone.js. The beat runs on its own clock, and your cadence only nudges the tempo, so the beat stays steady even when tracking jitters.
  • Tempo control: a rule-based controller handles safety: a floor at your starting pace, a ceiling at +15%, easing off when form slips, and holding when you're off the beat.
  • Per-person model: trained live in the browser. It's a weighted estimate of how much of each speed-up you follow, plus a 4-feature logistic regression (SGD) that predicts a form slip a few seconds ahead. It warm-starts from your past sessions and falls back to the rules until it has enough data.
  • Gemini coach: after the session ends, a Vercel serverless function sends a session summary to the Gemini API (only numbers, never video). It asks for structured JSON output that follows a fixed schema: summary, what went well, what to work on, trend across sessions, what the model learned, and next-session targets. The prompt keeps next session's targets close to this session's. Gemini never runs during the live session.
  • Privacy: video never leaves the browser.
  • Stack: Vite + TypeScript, Canvas charts, Vercel.

Challenges we ran into

  • Counting steps reliably from a side view at very different speeds. Slow marching kept getting missed until I switched to stride-based cadence and a longer window.
  • Measuring form without labels. Lift is scored against your own live range, and lean against a baseline it records right after the beat locks.
  • Stopping the controller from fighting the user: speeding up before form was measured, getting stuck at the floor, and false "off beat" readings.
  • Honest carryover scoring. Standing still now scores 0% instead of being skipped.

Accomplishments that we're proud of

  • Counted 40/40 steps at 80 bpm against a hand count
  • Pace held 91–103% in carryover checks in my own testing
  • Follow rate 57–81% across sessions
  • The whole live loop runs on-device in real time

What we learned

Keeping AI out of the real-time loop made everything simpler and safer. The beat needs to be dumb and steady, and the smart parts should advise, not drive.

What's next for Andante

  • Testing with physiotherapists to see whether the form cues match what they look for
  • A treadmill and running mode
  • A progress view across weeks
  • Better tracking in low light

Built With

Share this project:

Updates

Submission history