FireFIT

FireFIT turns Fire TV into an AI-powered fitness coach, using real-time computer vision to track reps, analyze form, provide on-screen corrections, and guide personalized workouts.

Comment

Inspiration

I'm a Software Engineering Intern, and when I discovered the Build, Ship, Shape Amazon Developer Hackathon I was excited by the challenge and the prizes. Most Fire TV homes have a phone within arm's reach but no camera on the TV, and most home workout apps just play a video and hope you're doing it right. I wanted the TV to watch your form using hardware people already own, so I built FireFIT TV and pushed hard to ship it before the deadline.

What it does

FireFIT turns your TV into an AI form mirror. You pick a program with the remote, scan the QR code on the TV, and prop your phone up. The phone tracks 33 body landmarks on device, counts reps only through full range of motion, and flags form problems. The TV shows your live mirrored video with a skeleton overlay, a rep ring, sets and rest, and depth and tempo for every rep across squats, push-ups and plank.

When you finish, the session is saved and you get an after-action report (what went well, what to fix, next time) that Amazon Polly can read aloud. An AI coach agent then reviews your history, explains the result, and decides your next workout: progress, hold, deload or technique focus. That plan becomes the "Recommended next" workout the next time you open the app. Profiles, onboarding, lifetime stats and streaks round out the experience.

Camera video never leaves your home network and never reaches AWS.

How I built it

  • Fire TV (Fire OS): a Kotlin WebView shell with a native audio engine, running a React + TypeScript 10-foot UI designed D-pad first.
  • Phone: a web page (no install) using MediaPipe Pose Landmarker and a deterministic TypeScript rep engine built on joint angles from world landmarks.
  • Phone to TV: WebRTC peer-to-peer over the home network. Video carries the skeleton, and a DataChannel carries reps and cues one way and remote commands the other.
  • AWS (defined in CDK, TypeScript):
    • Amazon API Gateway (HTTP API): single HTTPS entry point with explicit routes, CORS locked to the CloudFront origin, throttling, and per-room tokens for pairing rooms that expire after 15 minutes.
    • AWS Lambda (Node.js 22): ApiFn validates and stores workouts (Zod) and invokes CoachFn asynchronously so the TV never waits on a model call.
    • Strands Agents SDK (TypeScript) + Amazon Bedrock (Amazon Nova 2 Lite): the coach agent uses three tools (get_profile, get_workout_history, analyze_trends). Statistics are computed in code, not by the model; output is a Zod-validated plan that code clamps before saving.
    • Amazon DynamoDB: profile, workouts with per-rep depth/tempo/cues, running metrics, each AI recommendation with its tool trace and latency, and TTL pairing rooms.
    • Amazon Polly: reads the after-action report aloud on the TV.
    • S3 + CloudFront host the apps; CloudWatch Logs records each agent run.

AI-native, agentic development

FireFIT was built with an agentic AI-native workflow: I directed AI coding agents against a written spec (CLAUDE.md), with verification gates on real devices before each block was accepted, CI on GitHub Actions, unit tests for the rep engine, workout state machine, API and CDK stack, and a progress log and friction log kept in the repo.

Challenges I ran into

  • Getting low-latency phone-to-TV video working on real Fire TV hardware; I had to cap the sender bitrate to keep it smooth.
  • Making rep counting trustworthy: reps only count through full range of motion, and the plank needed different logic than squats and push-ups.
  • Keeping the AI coach honest with guardrails so targets match its decision and cues match the exercise.
  • AWS account onboarding friction (an SCP issue) that cost time early on.
  • Fitting a polished 10-foot UI, audio, profiles and reports into a hackathon timeline.

Accomplishments that I'm proud of

  • A full session (squats, push-ups, plank) verified end to end on a real Fire TV.
  • An agent whose decision changes the product: the next workout on the TV comes from the coach.
  • Privacy by design: video stays peer-to-peer; only structured workout metrics reach the cloud.
  • A signed v1.0.1 release with Fire TV launcher art, installable today.

What I learned

I learned how to design for the 10-foot experience and a D-pad, how to run on-device pose estimation in the browser, how to wire WebRTC between devices, and how to build a tool-using agent with Strands on Bedrock while keeping the math in deterministic code. I also learned how much faster I can ship when AI agents do the building and I own the spec, the verification and the decisions.

What's next for FireFIT TV

More exercises, automatic exercise detection, household profiles with Amazon Cognito, voice control, and publishing on the Amazon Appstore.

Try it

  • Fire TV (Fire OS 6/7/8): install Downloader from the Amazon Appstore, allow it to install unknown apps, enter tinyurl.com/firefitdev, install and open FireFIT, then scan the QR code on the TV with an Android phone (Chrome). Full steps and the adb path are in the README.
  • Code: https://github.com/jimmieldavisjr/firefit-tv
  • Demo video: https://youtu.be/iwzbAs5XmxM

Built With

Share this project:

Updates