RepCast

The coach on your TV that sees every rep. Prop up your phone and your Fire TV judges every squat, lunge and push-up — honestly — with live video, a 3D trainer and an AI coach's note.

Comment

Inspiration

Working out at home has one big gap: nobody is watching. Apps count reps you never did, a video can't tell a squat from a lunge, and a personal trainer costs more than most people can pay every week. Meanwhile almost every living room already has two things: a TV and a phone with a good camera. We wanted the TV, the biggest screen in the house, to become a coach that actually sees you, without buying a camera, signing up or sending your video anywhere.

What it does

RepCast turns any Fire TV and the phone you already own into a form coach.

  • Coach me on camera (12 moves): pick a move with the remote; scan the QR code with your phone (it opens a web page, nothing to install); lean the phone against a chair and step back. When you're in frame, the set starts by itself. The phone tracks your body, and the TV shows your live picture with your skeleton drawn on it, the count, and a verdict for every rep: clean, counted with a note, or no rep — and why ("Not low enough", "That's a split squat — this set is squats, feet side by side"). A coach's voice counts and tells you what to fix.
  • Honest by design: it never "butters" reps. It tells look-alike moves apart (squat vs lunge vs split squat, forward vs reverse lunge, step jack vs jumping jack) and checks them from the front, at 45° and from the side.
  • After the set: a receipt with every attempt, and Analyse with AI — a dashboard worked out on the TV plus a coach's note written by Amazon Nova on Amazon Bedrock from your numbers only (never your video), comparing you with last time and planning your next session.
  • Follow along (17 moves, no camera): a 3D trainer, a voice and a tempo beat that counts with you — planks and wall sits too.
  • For the whole family: profiles for up to 6 people, each with their own streak and a GitHub-style year of progress, all kept on the TV — no accounts.
  • Private: the video goes only from your phone to your TV over your own Wi-Fi, never to the cloud. If the picture can't get through (phone on another network), both screens say so, and reps still count.
  • Runs on the cheapest Fire TV: built and measured on a Fire TV Stick 3rd gen (32-bit Fire OS, about 1 GB of RAM): live 480p video with the skeleton at ~20 updates a second, voice and judging, using 135–200 MB of memory.

How we built it

  • Fire TV app: React Native for TV (react-native-tvos 0.86, new architecture) with Expo SDK 57, TypeScript. Every screen works with the D-pad; Play/Pause pauses a set. The live skeleton is drawn with native-driven Animated views (not React re-renders or SVG), which the 1 GB stick needed.
  • Phone camera page: plain TypeScript + Vite, served from GitHub Pages through repcast.netlify.app. MediaPipe Pose (Tasks Vision, WASM + GPU) runs entirely on the phone; our engine judges each rep there.
  • Open-source rep engine — repcast-movement (MIT): turns pose landmarks into reps, holds and verdicts: joint angles with adaptive smoothing, footwork (which foot moved, forward/back/across), stance width, posture, and per-move rules. 73 unit tests; tuned on real footage through a replay bench.
  • Phone → TV: pairing over a 4-digit code; WebRTC peer-to-peer video on the home Wi-Fi (H.264, hardware-decoded on the stick); pose over a WebRTC data channel; events and the TV's commands (which move, which set, rest, pause) over AWS AppSync Events. The TV leads; the phone follows.
  • AWS backend — repcast-backend (MIT, SAM): AppSync Events (live channel), API Gateway HTTP API + Lambda + DynamoDB (pairing sessions with a 2-hour TTL, AI-note allowances), Amazon Polly (all 120 coach lines, recorded in CI and bundled — Fire TV's own TTS is silent for apps), Amazon Bedrock (Nova Lite via Converse with a forced tool call for structured JSON; the facts are computed in code so the model never does arithmetic), SSM Parameter Store, AWS Budgets. Deployed only by GitHub Actions with OIDC — no stored AWS keys.
  • 3D trainer: Mixamo animations rendered in Blender on the app's own background colour, so clips sit seamlessly in the UI.
  • Testing: a replay bench that plays real gym/office clips and public exercise videos through the exact phone pipeline; 3D renders of every judged move from three angles; a side-by-side harness that logs phone and TV counts; and end-to-end runs on two real Fire TV Sticks.

Challenges we ran into

  • Fire TV's text-to-speech is silent for apps (no error) — every voice line is pre-recorded with Amazon Polly instead.
  • A 1 GB, 32-bit stick: a 64-bit-only APK won't install on the 3rd gen even though its CPU is 64-bit; re-rendering a live skeleton in React saturated the CPU (fixed with native-driven views); we measured memory and jank on the device after every big change.
  • Judging honestly is hard: MediaPipe swaps left and right ankles facing the camera, side-on split squats look like lunges, and phones run anywhere from 7 to 30 fps. Each fix came with a test and replayed footage.
  • Bedrock access: our own account stayed in "account verification" (Bedrock "Operation not allowed"), and Claude on Bedrock needed Marketplace permissions — the notes are written by Nova Lite through a second account we have access to, with its keys in Parameter Store. In our final test about 1 Nova call in 4 hung silently; the backend now cuts a call off at 7 s and retries (10/10 notes written after the fix).
  • All 23 snags are in our friction log, each with a workaround and a suggestion.

Accomplishments that we're proud of

  • Live phone video on a 1 GB Fire TV Stick, with the skeleton aligned and ~33 ms behind the picture (Fire TV Stick 4K).
  • A coach that refuses wrong reps out loud: a split squat during a squat set is a "no rep — that's a split squat", every time, on real gym footage.
  • A setup a non-technical person can do: scan, lean the phone, step back — the set starts itself.
  • The rep engine stands on its own as an MIT library any app can use.

What we learned

The TV is a great coach screen but a tiny computer: design for the stick first. Honest feedback beats generous feedback — people trust a "no rep" that explains itself. And computing the facts in code, then letting the model write, gives AI notes that are friendly and correct.

What's next

Publish on the Amazon Appstore; more moves (lunges with weights, pull-ups, yoga holds) with the same honesty; a Vega OS version; multi-person sets for families; and a coach that adapts the plan week by week from your history.

Built With

Share this project:

Updates

Submission history