Inspiration

During the pandemic, most of us shifted our workouts to the living room, following fitness channels on the big screen. But streaming videos on TV has a glaring issue: no one is there to check your form. Bad posture in basic movements like squats, lunges, or planks doesn't just waste effort—it directly leads to back and knee injuries. We realized millions of people already have powerful Fire TV hardware connected to their main living room displays. We wanted to see if we could turn a Fire TV stick and an inexpensive USB webcam into a live, interactive personal trainer that keeps you safe and guides your posture in real time.

What it does

FitVision TV turns your Fire TV into an interactive, real-time workout coach using a standard USB camera plugged into your TV: Real-time Form Tracking: Draws a live skeletal overlay over your video feed right on the TV screen. Angles stay green when your posture is spot-on and flash red when your knees, back, or elbows fall out of alignment. Contextual Voice Coaching: When bad form is detected, the app uses AWS Bedrock to process joint angles and instantly gives natural verbal corrections through the TV speakers—like "Lower your hips slightly" or "Keep your back straight." On-Device & Private: Frame processing runs completely locally on the Fire TV hardware, ensuring your camera feed never gets uploaded to the cloud or saved anywhere. Session Summaries: Tracks your total reps, movement accuracy scores, and posture trends per exercise, saving stats to your profile so you can monitor form improvements over time.

How we built it

Front-End & Video Input: Built the Fire TV app using React Native, utilizing standard Android OTG APIs to capture live video feeds from a USB camera. On-Device Posture Engine: Integrated MediaPipe Pose locally within the app to run keypoint estimation at 24+ FPS on-device, ensuring strict privacy without cloud video processing. AWS Bedrock Coaching: Triggered an AWS Lambda function whenever joint posture anomalies occurred. AWS Bedrock (Claude 3.5 Sonnet) evaluated mathematical joint coordinates to formulate natural coaching advice. Voice Synthesis: Pushed Bedrock’s coaching output through Amazon Polly to generate crisp, real-time voice feedback directly through the TV speakers.

Challenges we ran into

Frame Rate Throttling on Fire TV Hardware: Running MediaPipe pose detection directly on low-power TV hardware alongside high-definition video playback initially dropped screen refresh rates below 12 FPS. We resolved this by downsampling raw camera feeds specifically for the inference engine while keeping the visual video stream rendered at full resolution. Camera Handling via USB OTG: Managing external USB webcam streams through Android OTG APIs required handling custom hardware permissions, video aspect ratio squishing, and frame orientation adjustments across different TV displays. Latency in Bedrock Audio Triggers: Direct API calls to AWS Bedrock for every detected frame created response lags and rate-limit hits. We implemented a local throttle buffer that batch-analyzes joint angle variance and triggers Bedrock only when a posture error persists for more than 1.5 seconds.

Accomplishments that we're proud of

Sub-100ms Inference on Low-Power TV Hardware: We successfully optimized MediaPipe Pose to run locally on Fire TV hardware, giving users instantaneous skeletal overlays and visual cues without lag. Privacy-First Architecture: Built a completely localized video pipeline where live webcam feeds are processed entirely on-device, sending only zero-PII joint coordinate vectors to the AWS cloud. Smart Voice Feedback Loops: Seamlessly coupled computer vision joint tracking with AWS Bedrock and Amazon Polly to deliver contextual, natural verbal feedback that feels like working with a real personal trainer. TV-First Multimodal UI: Designed an intuitive, glanceable TV interface that keeps key exercise metrics visible during workouts without distracting from the main video feed

What we learned

Optimizing Vision Models for Low-End ARM Hardware: We gained deep hands-on experience tuning TensorFlow.js and MediaPipe parameters to extract maximum performance out of low-power TV chips without overheating or dropping frame rates. Designing for 10-Foot UI/UX: Designing for a TV screen is completely different from mobile or web. We learned how to prioritize large, high-contrast visual cues, bold typography, and ambient audio feedback that users can process from 10 feet away without breaking focus during a workout. Smart Offloading with AWS Bedrock: We learned how to balance client-side and cloud workloads—using the local device strictly for fast mathematical posture calculations, and calling AWS Bedrock only when deep reasoning is needed to turn raw angle data into human-like advice.Optimizing Vision Models for Low-End ARM Hardware: We gained deep hands-on experience tuning TensorFlow.js and MediaPipe parameters to extract maximum performance out of low-power TV chips without overheating or dropping frame rates. Designing for 10-Foot UI/UX: Designing for a TV screen is completely different from mobile or web. We learned how to prioritize large, high-contrast visual cues, bold typography, and ambient audio feedback that users can process from 10 feet away without breaking focus during a workout. Smart Offloading with AWS Bedrock: We learned how to balance client-side and cloud workloads—using the local device strictly for fast mathematical posture calculations, and calling AWS Bedrock only when deep reasoning is needed to turn raw angle data into human-like advice.