Couch Coach

Your phone is the camera, your Fire TV is the coach: it draws your live skeleton, counts reps, scores your form, and an AI coach on AWS that remembers you speaks after every set.

Comment

Inspiration

The biggest screen in the house is the best place to watch yourself move, but a Fire TV has no built-in camera, and a phone screen is too small to read from three metres away. Couch Coach splits the job. Your phone is the camera. Your Fire TV is the mirror and the scoreboard. A coach in the cloud talks to you after every set, and it remembers how you did last time.

What it does

  1. Pick a Member with the remote. The Fire TV app lists the people in the household. D-pad to a name, press Select.
  2. Pair by QR. The TV shows a QR code. Scan it with any phone, allow the camera, and you are paired. No account, no app install on the phone.
  3. Work out. Prop the phone up two to three metres away. The phone runs pose estimation on the device and sends only body landmarks, never video. The TV auto-frames you and draws your live skeleton large, counts squats and jumping jacks, keeps a running Form Score, pulses the legs mint on a clean rep, and highlights the limb and names the fault (for example "Knees caving" or "Feet not wide") when a rep goes wrong.
  4. End the set. An AI coach (Claude Haiku 4.5 on Amazon Bedrock) reads this set plus that Member's recent history and writes a short Cue. Amazon Polly speaks it on the TV about one to two seconds after the set ends, and the same text appears as a caption on the TV and on the phone.
  5. Recap. End the session and the TV shows sets, reps, Form Score per set and the Coach's note.

The Coach "remembers" because every set is saved per Member in DynamoDB and the next Cue is written against the previous set of the same exercise (improved, same or worse). The trend is computed in code, not left to the model.

Why it fits the Fire TV track

The brief's own "creative" examples are a computer-vision fitness app that uses a camera, and multi-modal UX. Couch Coach is both:

  • Computer vision with a camera: MediaPipe Pose Landmarker on the phone camera, plus our own open-source rep counter and form scorer (pose-reps).
  • Multi-modal on a TV: the phone camera sees you, the phone doubles as a controller (pick the exercise, start and end the set), the Fire TV remote D-pad drives the TV UI (members, pairing, dialogs), and the Coach speaks back with Amazon Polly. Spoken input is not built yet; see What's next.
  • Designed for the 10-foot screen: a "Lamplight" design system with large type (26 px minimum), WCAG AA contrast targets, a visible D-pad focus treatment on every control, no colour-only signals, and the spoken Cue always captioned.

How we built it

Phone (web app, Vite + TypeScript, served by CloudFront over HTTPS): MediaPipe Pose Landmarker (lite model, GPU delegate with CPU fallback) runs in the browser. pose-reps counts reps and scores form on the phone, and each pose frame carries the live squat depth and phase. Landmarks are batched and sent over a WebSocket to AWS AppSync Events. Video never leaves the device.

Fire TV app (Fire OS, sideloaded APK): React Native for TV (react-native-tvos 0.81 on Expo 54), forked from Amazon's react-native-multi-tv-app-sample and reduced to four screens: Members, Pair, Live, Recap. A jitter buffer plays the landmark stream back smoothly; an auto-framing camera zooms (1 to 3x, with hysteresis and easing) so you fill about 80% of the stage even standing far back; an imperative SVG loop draws the skeleton without re-rendering React per frame.

Cloud (one AWS CDK stack, us-east-1):

  • AWS AppSync Events carries pose frames, rep and set events and the Coach channel between phone and TV. We wrote a small shared WebSocket client (no Amplify).
  • Amazon API Gateway (HTTP API) + AWS Lambda (Node.js, arm64): a session function creates a Session and a signed phone token (HMAC key in AWS Secrets Manager); a coach function validates the token, enforces per-session and daily caps, loads the Member's last five sets, calls Amazon Bedrock Converse (Claude Haiku 4.5), synthesises the Cue with Amazon Polly (neural), stores the MP3 in Amazon S3 behind a short-lived presigned URL, saves the set, and publishes the Cue back over AppSync Events.
  • Amazon DynamoDB (one table, on demand, TTL) holds Members, Sessions and Sets: this is the Coach's memory.
  • Amazon CloudFront + S3 serve the phone web app.
  • Amazon CloudWatch, SNS and AWS Budgets raise alarms on Bedrock invocations, Polly characters, Lambda errors and spend.

Open source: rep counting and form scoring live in a separate MIT library with zero runtime dependencies, pose-reps (v0.2.0).

Quality: more than 550 automated tests in the app repo plus 190 in pose-reps (99% statement coverage), a 24-call Bedrock eval for the Coach's Cues, and a STRIDE security review (SECURITY.md) whose two high-severity findings were fixed and deployed before submission.

Challenges we ran into

  • The phone gave us the wrong picture. On an iPhone held upright, Safari returned a landscape camera frame even though the page asked for portrait. The body filled a third of the frame, landmarks collapsed, reps were over-counted and faults were false positives. We added a retry with an exact 9:16 aspect ratio, framing hints ("Step back", "Come closer") and a frame gate that rejects collapsed skeletons.
  • Facing the camera hides the squat. Seen head-on, your knees come toward the lens, so the 2D knee angle barely changes even when you sit deep, and v0.1 missed reps. pose-reps 0.2.0 measures how far the hips drop, in shin lengths, against your own standing height, smoothed with a frame-rate-independent One-Euro filter. On our real recording the count went from 6 to 8, at every frame rate from 10 to 30 fps.
  • The Coach contradicted the data. One Cue praised "feet staying wider" while the app had flagged narrow feet on 8 of 10 reps. We moved the decisions (main fault, trend, when to use the name) into code, kept the model for wording only, and built an eval: the pass rate went from 8 of 24 to 24 of 24. The checks are our own keyword rules, so they guard against regressions rather than prove the Cues are good. Filming also caught the Coach calling yesterday's set "earlier today"; the day comparison now uses calendar days.
  • A public realtime channel is not private. Our security review showed that the public AppSync API key could subscribe to every Session with a wildcard and forge Cues. We added subscribe handlers that reject wildcards, moved Coach messages to a namespace only the Lambda can publish to, made the TV play audio only from our bucket, and capped spend.
  • We missed one latency target. Phone-to-TV p95 latency is about 330 to 400 ms against our 300 ms goal. The skeleton still looks live because the TV buffers and interpolates, at the cost of drawing you slightly behind real time.
  • Scope cuts, stated plainly. AgentCore Runtime and Memory were planned and cut (ADR 0004); the Coach uses Bedrock Converse plus DynamoDB history instead. The Vega SDK was not used; the TV app is a Fire OS APK.

Accomplishments that we're proud of

  • A complete loop that works end to end on a real Fire TV Stick (3rd generation, Fire OS 7.7.1.7): QR pairing, phone camera, live auto-framed skeleton, reps and form score, a spoken and captioned Cue one to two seconds after the set, and a recap.
  • Pose estimation stays on the phone, so the privacy story is simple: only body landmarks leave the device.
  • A reusable open-source library with zero runtime dependencies, 190 tests, CI and real-recording regression tests.
  • A Coach that is checked, not just prompted: deterministic checks, a committed eval report, and code that owns the facts.
  • Security reviewed and hardened before submission, with the findings, fixes and residual risks published in the repo.
  • The demo video was filmed from the devices themselves: the Fire TV records its own screen over adb, and a timeline of every rep and Cue syncs it with the phone's screen recording.

What we learned

  • On a phone, the camera frame you get is not the frame you asked for. Check videoWidth and videoHeight before trusting any geometry.
  • A 2D joint angle lies when the limb points at the camera; measure something that survives foreshortening.
  • With an LLM in the loop, put facts in code and wording in the model, then write an eval that can fail.
  • A listed model or an "ACTIVE" inference profile is not the same as one you can invoke. Probe before you build on it.
  • Realtime pub/sub needs per-channel authorisation designed in from the start; unguessable channel names are not enough.

What's next for Couch Coach

  • Real per-Session authorisation on AppSync Events (an AWS_LAMBDA authorizer on the HMAC phone token), so the public key can go away.
  • Voice commands on the remote and Alexa ("start squats", "end set"). Today the Coach speaks but does not listen.
  • A live depth gauge on the TV (the phone already sends depth with every frame), and more exercises, lunges first, tuned on labelled real recordings.
  • Households with login, so Members are not shared across installs.
  • An AgentCore-backed Coach, if long-term memory proves worth the extra moving parts.
  • Publish pose-reps to the npm registry (today it installs from the git tag).

Built With

Share this project:

Updates

Submission history