Spotter

A Fire TV fitness coach that uses your phone as the camera. The phone counts your squats and checks depth, the TV shows each rep live, and you get a coaching summary after every set.

Comment

Inspiration

Workout videos on TV play at you and never check what you do. A phone propped six feet away has a screen too small to read mid-squat. I wanted the TV to count my reps, tell me when a squat was too shallow, and sum up the set at the end. Fire TV sticks have no camera, but almost everyone has a phone.

What it does

Open Spotter on a Fire TV and it shows a QR code. Scan it with an iPhone, prop the phone up side-on, and start squatting.

The phone runs pose tracking on the device and measures your knee angle on every frame. A rep counts when your knee goes below 100 degrees and you stand back up. If you dip but stop short of 100 degrees, Spotter counts a half rep and says "Go lower" out loud. A rep below 85 degrees gets "Great depth".

The TV shows the rep count, a depth bar that turns green at parallel, your current knee angle and the latest cue, in text large enough to read from across the room. Tap Finish, and the server builds a summary from your rep log, which holds the depth and tempo of each rep, the half reps and the set length. The TV shows the summary and the phone reads it aloud. Tap "Start a new set" to go again.

Video stays on the phone. Only the numbers go to the server.

How I built it

Spotter has three parts that share one TypeScript package for message types.

The TV app is React Native 0.83 on Vega OS. I generated it from the Vega SDK 0.24 helloWorld template and ran it on the Vega Virtual Device. It lays out on a 1920×1080 canvas and scales to the screen. A pure JavaScript encoder produces the QR code and plain React Native views draw it, so the app needs no native module.

The phone app is a Vite and TypeScript web page. It runs MediaPipe Pose Landmarker in Safari, picks whichever leg the camera sees better, smooths the angle, and feeds a small state machine that tracks full reps, half reps, depth and tempo. The Web Speech API speaks the cues.

The server is a Node WebSocket relay. It pairs a TV and a phone with a 4-digit room code, puts the Mac's LAN address in the QR code, and forwards live state to the TV. For the set summary, it calls Amazon Bedrock's Converse API with Amazon Nova Lite, and it falls back to a rule-based coach if that call fails or takes longer than 8 seconds. On deadline day AWS blocked my account from Bedrock with "Error 002", which I logged in the friction log, so the demo shows the rule-based coach. Anyone who runs the server with Bedrock access gets the Nova Lite summary with no code changes.

Vitest and Jest tests cover the rep counter, the coaching logic, the Bedrock call and its fallback, and the TV screen transitions.

Challenges I ran into

Setting up Vega took longer than writing the first feature. The recommended starter sample failed to build on SDK 0.24 for three reasons. One script used syntax that Yarn 4 rejects, three Babel plugins that Reanimated needs were missing, and the manifest lacked the OS version that 0.24 requires. I fixed all three, and Amazon merged the fix as AmazonAppDev/react-native-multi-tv-app-sample#130.

iOS only gives camera access to HTTPS pages, so the phone app serves over a self-signed certificate on the LAN and proxies the WebSocket through the same origin. iOS also blocks speech unless the page starts it inside the first tap, before any async work.

The Virtual Device lays out at a lower resolution than the 1080p I designed for, and my first pairing screen overflowed. Scaling a fixed 1920×1080 canvas fixed it at every screen size.

Accomplishments that I'm proud of

Spotter runs on hardware people already own, a phone and a Fire TV, and the video never leaves the phone. Half-rep detection tells you something a workout video can't. My fix was also merged into their sample app while I was building this.

What I learned

I learned how Vega apps fit together, including manifests, OS module versions, .vpkg packages and the Virtual Device. I found that MediaPipe in mobile Safari tracks a full body at a usable frame rate. On a TV, one big number and a few words work better than a busy screen.

What's next for Spotter

I want to add lunges, push-ups and planks, which use the same joint-angle approach. After that come multi-set workouts with rest timers, Alexa voice control on Fire TV so you can say "Alexa, finish set", and a weekly progress screen on the TV.

Built With

Share this project:

Updates

Submission history