AuraForm Omni

Inspiration

Traditional home workouts often lack the immediate feedback and personalized guidance that people receive from an in-person trainer. Following workout videos can make it difficult to know whether an exercise is being performed correctly, when to take a break, or how to adapt a session to individual needs.

We built AuraForm Omni to explore a more interactive approach to home fitness: transforming a TV into an intelligent workout companion that combines computer vision, real-time feedback, and context-aware coaching. Our goal is to make workouts more engaging and responsive while keeping the experience accessible from the comfort of home.

What it does

AuraForm Omni turns a TV into an AI-assisted fitness coaching experience. It combines camera-based pose analysis with workout tracking, coaching cues, and recovery guidance.

Key features include:

• Real-time pose analysis: Uses computer vision and MediaPipe landmarks to analyse squat movements and evaluate aspects of exercise form.
• Automatic repetition counting: Tracks squat repetitions and provides workout statistics.
• Form feedback: Identifies supported movement and posture issues and provides coaching cues.
• Adaptive recovery guidance: Combines deterministic safety rules with available workout and vital-sign context to determine when recovery guidance is appropriate.
• TV-first experience: Provides a dashboard, active workout screen, recovery screen, and session summary, with navigation designed for a remote control.
• Phone-camera pairing: Lets users open a camera page on their phone through a QR code and stream frames to the computer-vision service.
• Extensible AI integrations: Supports optional Amazon Bedrock-powered coaching and an optional Bee integration for additional context and recovery tasks.

The system is designed so that core workout analysis and local coaching can operate without depending on a live LLM response.

How we built it

AuraForm Omni is organized as a modular system with a TV frontend, a computer-vision service, and an AI-agent orchestration layer.

TV frontend: We built the interface with React Native and Expo, focusing on TV navigation, workout states, coaching feedback, recovery experiences, and session summaries.

Computer vision: A Python FastAPI service uses OpenCV and MediaPipe to process camera frames, extract body landmarks, evaluate squat-related movement metrics, and track repetitions. It supports local and remote camera input, along with synthetic data for development and demonstrations.

Agent orchestration: A Node.js service coordinates telemetry, workout state, vital-sign context, coaching decisions, and UI updates. A shared state hub and session tracker help maintain context across the workout.

Model Context Protocol (MCP): MCP tools expose workout telemetry, vitals, session history, and UI-control capabilities to the agent layer.

Amazon Bedrock: Bedrock is an optional integration for AI-generated coaching and decision support. A deterministic local policy remains responsible for enforcing the application's recovery rules.

Integrations and infrastructure: The project includes optional Bee-related functionality, Docker Compose configuration, Envoy routing, Terraform for a Bedrock invocation IAM policy, and automated testing and demonstration scripts.

This modular architecture lets the computer-vision pipeline, coaching logic, and TV experience evolve independently.

Challenges we ran into

One of the biggest challenges was coordinating multiple real-time components without making the workout experience dependent on a single service. Camera frames, pose landmarks, repetition updates, vital signs, agent decisions, and TV events all need to remain consistent.

We addressed this by separating the computer-vision pipeline from orchestration and UI rendering, using shared state and explicit workout states to coordinate the experience.

Another challenge was balancing AI flexibility with predictable recovery behaviour. A language model can generate useful coaching, but safety-related decisions should not rely entirely on free-form model output. We therefore designed a deterministic local policy to govern recovery decisions, with the LLM providing an optional coaching layer.

We also had to account for the differences between TV and phone interfaces, particularly camera pairing, remote-control navigation, and the availability of hardware integrations across platforms.

Finally, testing with synthetic data and automated checks is not the same as validating real-world exercise analysis. Camera positioning, framing, lighting, and missing or noisy landmarks can affect the quality of movement estimates. These remain important areas for further testing.

Accomplishments that I'm proud of

• Built a connected fitness prototype spanning a TV application, a computer-vision service, and an agent orchestrator.
• Implemented camera-based squat analysis, repetition tracking, and form-feedback capabilities.
• Created a complete workout interface flow covering the dashboard, active exercise, recovery, and session summary.
• Designed a local-first coaching path that does not require every decision to go through a cloud model.
• Integrated MCP tools to make workout state and UI capabilities accessible to the agent layer.
• Added phone-camera pairing to support a flexible camera setup.
• Created supporting infrastructure, automated checks, and scripts for testing important workflow and recovery scenarios.

We are proud that AuraForm Omni is not just a workout screen or a standalone pose detector, but an attempt to connect perception, decision-making, and an interactive fitness experience into one system.

What we learned

We learned that building an AI-powered experience involves much more than connecting a model to an interface. Reliable state management, predictable fallback behaviour, clear service boundaries, and honest evaluation are equally important.

We also learned that computer-vision outputs need careful interpretation: a detected landmark or calculated angle is not, by itself, proof that an exercise is safe or clinically correct. Real-world testing is essential before making stronger claims about accuracy or injury prevention.

Finally, platform constraints matter. TV navigation, phone-camera connectivity, Bluetooth access, cloud credentials, and optional integrations each have different requirements. Designing around these constraints early makes the overall system more robust.

What's next for AuraForm Omni

Our next steps are to validate the experience on the target Fire TV environment, improve camera framing and movement-analysis robustness, and expand automated testing with representative real-world workout footage.

We also want to develop more exercise-specific analysis pipelines, improve personalization using reliable workout history and available vital-sign data, and make the coaching experience more responsive to individual needs.

On the integration side, we plan to validate live Amazon Bedrock and supported wearable connections in their intended environments, improve observability across services, and strengthen deployment and security configuration.

Our long-term vision is to make AuraForm Omni a more adaptive, accessible, and trustworthy fitness companion that brings intelligent coaching into the living room.

Built With

• amazon
• amazon-web-services
• bedrock
• context
• docker
• expo.io
• fastapi
• mcp
• mediapipe
• model
• native
• node.js
• opencv
• protocol
• python
• react
• terraform
• typescript
• websockets