Spotter

A talking workout coach that watches your form with OpenCV 5, tells you how to fix it, checks that you did, and changes the plan when your body needs it.

Comment

Inspiration

Most people train at home alone, following videos. But a video can't see you: it can't tell you that your hips are sagging in a plank, and it keeps going while your form falls apart. We wanted a coach that actually watches you, talks to you, and adapts, using nothing but a phone.

What it does

Spotter is a talking workout coach. You pick Lower body or Upper body (4 exercises × 60 s, 30 s rest). The top of the screen shows an animated coach doing the move; the bottom shows you, with your skeleton drawn on top and wrong joints in red.

  • Camera check first (active perception): before each move, the coach checks light, blur, distance and camera angle with OpenCV, and tells you what to fix ("It's a bit dark for me, could you turn on a light?"). The set starts only when she can actually judge the exercise.
  • Perceive → decide → act → re-check: OpenCV 5 finds 33 body keypoints; geometry turns them into faults (hips low, hips high, knees down…). A fault must persist before the coach reacts, so one bad frame never triggers a false alarm. She gives a spoken cue, then checks the next frames: praise only when the fix is real.
  • Plan changes with human approval: if a correction keeps failing, the agent picks a tool (easier variant, shorter set, or ask the user). No answer in 10 s means a safe pause, never "push through".
  • Notices stops and the wrong exercise: stop in the middle and the timer pauses ("You held 32 seconds. Continue or rest?"), then resumes when she sees you back. Select plank but do squats, and she recognises it and asks whether to switch.
  • You talk to her: "pause", "skip", "end", "am I doing it right?" (answered from what the camera sees), "how long is left?", and "it hurts" stops the set at once.
  • Upload a video: the product is made for live practice; uploading a recording runs the same loop, which we use for demos and evaluation.
  • Every decision is written to a trace (in the app, downloadable, and in CloudWatch Logs).

How we built it

  • OpenCV 5 (cv.dnn) runs the OpenCV Zoo MediaPipe person detector and pose model (33 keypoints), plus warpAffine alignment, CLAHE for dark frames, zoom re-detection for small or far people, and cv.Laplacian for blur.
  • AWS Lambda (Python 3.13, Function URL over HTTPS) runs the whole vision pipeline and the agent; the server is stateless (the agent state travels with each request). Amazon Polly gives the coach her voice; CloudWatch Logs stores one JSON line per agent event.
  • The decision layer is built for Amazon Bedrock (Amazon Nova, Converse API with tool use). Bedrock is currently blocked on our AWS account ("Operation not allowed" for every model, even in the console), so each decision automatically falls back to a tested deterministic policy that chooses among the same tools, and the trace records who decided.
  • Front end: vanilla JS, browser camera and speech recognition, an original animated coach drawn in code.

Challenges we ran into

  • OpenCV 5 found no person at all: its new DNN engine returns the detector outputs in a different order than 4.x. We match them by tensor shape.
  • The coach praised users who walked away: a vanished fault counted as "fixed". Now only a clean frame counts.
  • Real rooms are hard: far away, accuracy was 64 %; in a dark room, 67 %. Zoom re-detection and CLAHE raised them to 81 % and 83 %, and the camera check now refuses those conditions.
  • A failed test: we uploaded a plank video labelled as squats, and the coach said "Form OK". We added exercise recognition from pose signatures.
  • Cloud blockers: Bedrock and CloudShell were blocked on a new account, so we deployed through the console and S3, then made the function load its latest code from GitHub on each cold start.

Accomplishments that we're proud of

  • A real closed loop where what the camera sees changes what the coach does next, shown frame by frame in the trace.
  • 74/74 plank frames correct on a labelled real video; 0 false wrong-exercise alarms on real plank, squat and lunge videos; 35–44 ms per frame for OpenCV 5 on Lambda.
  • An evaluation that keeps the failures: each one is documented with its cause, its fix and the result after.
  • A coach that feels human: she waits while you set up, varies her sentences, listens, and never pushes through pain.

What we learned

Testing on clean footage is not enough: darkness, distance, blur and the user simply walking away broke things a clean test video never showed. We also learned that a good agent is mostly about re-checking its own actions and knowing when to ask the human.

What's next for Spotter

Many more exercises with their own form checks, full programmes built around a goal (strength, mobility, back health) that progress week after week, a coach that remembers your usual faults, more languages, tests on many more people, bodies and rooms, and Amazon Nova making the plan decisions as soon as Bedrock is enabled on the account.

Built With

Share this project:

Updates

Submission history