GhostTrainer

GhostTrainer turns at-home workouts into an unskippable ritual: Ring summons your AI coach to the porch, Alexa calls you into the session, and a Fire TV mirror tracks every rep and squat in real time.

Comment

GhostTrainer: Cross-Device Ambient AI Fitness Coach

GhostTrainer turns at-home workouts into an unskippable ritual: Ring summons your AI coach to the porch, Alexa calls you into the session, and a Fire TV mirror tracks every rep and squat in real time.

What Inspired Us

Most home workout plans fail for the same reason: nothing happens at the moment of decision. The alarm goes off, the phone buzzes, and the easiest option wins. We wanted to build a system that shows up where the user already is, which means the doorbell, the hallway speaker, and the living room TV, instead of asking them to open yet another app.

The Amazon ecosystem already has these devices in many homes. The question was whether they could work together as one coaching loop rather than as isolated gadgets.

What It Does

GhostTrainer runs a four-stage ambient loop:

  1. Doorstep summon (Ring simulator). At the scheduled time, a doorbell event fires and a virtual coach "arrives" on the porch.
  2. Conversational bridge (Alexa+ via MCP). Alexa confirms readiness and starts the session through a Model Context Protocol server exposing tools such as get_workout_schedule, start_workout_session, report_workout_metrics, and get_ai_coach_recovery_brief.
  3. Smart mirror (Fire TV). A 10-foot React and Tailwind UI runs MoveNet pose estimation in the browser through TensorFlow.js, tracking reps, squat depth, and knee alignment. It is navigable with the standard D-pad keys.
  4. Agentic feedback (AWS Bedrock). Session telemetry goes to Claude through Bedrock. The coach gives short in-workout cues and produces a post-workout recovery brief.

If AWS credentials are not configured, the system falls back to deterministic rule-based coaching so the demo works offline.

How We Built It

Architecture. The project is a monorepo with four services: a FastAPI Ring simulator, a FastAPI Streamable HTTP MCP server for Alexa+, a Fire TV web app, and a shared bedrock-coach Python package. Docker Compose and a one-command start_demo.sh launch the full stack locally.

Backend. We used Python 3.11+, FastAPI, and Pydantic v2 with async endpoints for webhook and MCP traffic. The Bedrock client is an async wrapper that returns typed responses and degrades gracefully when the model is unavailable.

Pose and rep logic. MoveNet provides 17 keypoints per frame. Squat depth is derived from the hip-knee-ankle geometry. For a simple view of how the score is computed, the recovery brief uses this heuristic in the offline path:

$$ S = \max\left(40,\ \min\left(100,\ \bar{d} \cdot \frac{R_{\text{done}}}{R_{\text{target}}}\right)\right) $$

Here, d-bar is the average depth percentage across reps, R_done is the number of completed reps, and R_target is the planned target. The floor of 40 keeps a partial session from reading as a failure, and the cap at 100 keeps the score within the schema.

Testing. Unit tests cover the Bedrock coach (schema round-trips, fallback behaviour, and invalid JSON handling) and the Ring and MCP services. A scripted end-to-end flow (scripts/demo_flow.py) drives the whole loop without a camera.

Camera-free demo mode. We built a synthetic MoveNet-style kinematic engine that generates 17-keypoint poses with injected form flaws. This lets judges, CI, and headless environments run the full biomechanics and voice-coaching path without a webcam.

What We Learned

  • Schema drift is silent. The frontend sent detected_issue while the telemetry model expected form_issue. Pydantic ignored the unknown field by default, so the coach received no flaw data and generated generic prompts. We now treat ingestion schemas as strict during integration testing.
  • Fallbacks can hide failures. Graceful degradation kept the workout running, but it also masked HTTP 500 errors (for example, depth values above 100 percent) during UI testing. The UI looked healthy while the backend was failing. Fallbacks need visible telemetry next to them.
  • Synchronous SDKs block async servers. A blocking boto3 converse call inside an async def endpoint starved webhook handling during bursts of rep telemetry. Wrapping the call with asyncio.to_thread or using an async client is required.
  • Browser and TV runtimes lag. CanvasRenderingContext2D.roundRect is missing on older Fire OS builds, so we added feature detection with a rect fallback. Camera access also varies by device, which is why the synthetic camera path exists.
  • Latency budgets are tight. A voice-style coaching cue needs to arrive well under a second to feel conversational. That pushed us toward short prompts, capped output tokens, and structured JSON responses.

We documented each of these in docs/FRICTION_LOG.md, with concrete recommendations for Amazon's developer tooling.

Challenges

1. Cross-device timing. The loop depends on asynchronous events: a Ring webhook, then an Alexa MCP call, then a Fire TV poll of /session/status, then Bedrock. Any one of these can be slow or unavailable. Webhook dispatch uses a short timeout and logs failures instead of crashing, and the coach falls back to rule-based cues when Bedrock is unreachable, so the workout keeps running. Retries with backoff are still on the to-do list.

2. Real-time pose on TV hardware. Browser-based MoveNet on low-power TV chipsets is tight. The target is a smooth 30 FPS, which means frame skipping and a lightweight MoveNet variant. We also had to avoid decoding multiple video streams at once, since the Ring picture-in-picture feed and the live camera compete for GPU memory.

3. No hardware in CI. Reviewers and automated pipelines have no webcam or UVC device, so getUserMedia fails with NotFoundError. The synthetic pose engine solved the testing problem and became a demo feature.

4. Keeping the coach grounded. LLM output is only useful if it matches the actual form data. Strict schemas, typed responses, and the rule-based fallback keep the coaching tied to measured telemetry.

5. Scope under a deadline. We cut anything that did not serve the core loop. The priority was a demo that runs end to end, not a complete fitness product.

What's Next

  • Replace the rule-based score with calibrated, per-user baselines.
  • Move the camera path to a device-tested Fire TV build, not only the browser.
  • Add prompt caching to the Bedrock path and measure first-byte latency against the target.
  • Connect real Ring and Alexa+ endpoints once developer access is available.

Built With

Python 3.11, FastAPI, Pydantic v2, Model Context Protocol (Streamable HTTP), React, Tailwind CSS, TensorFlow.js, MoveNet, AWS Bedrock (Claude), Docker Compose, pytest. Released under the MIT License.

Built With

Share this project:

Updates

Submission history