Amazon Steady

A Fire TV coach that helps older adults avoid falls: 28 tracked exercises planned around your health. Your phone is the camera; the TV coaches, counts, adapts and checks on you.

Comment

About the project (the main description field)

Inspiration

Every year, one in four adults over 65 falls. That's 14 million people in the US, and falls are the leading cause of injury death at that age. The frustrating part is that we already know what works. Simple strength-and-balance programmes like Otago cut falls by about a third. Yet a year in, only about a third of people are still doing their exercises. The moves are simple; doing them alone, with no feedback and no sense of progress, is not.

The TV is the biggest, most familiar screen an older adult owns, and Fire TV is already in their living room. But most Fire TVs have no camera, and asking a 76-year-old to buy and mount a webcam is a non-starter. Everyone has a phone, though.

What it does

Amazon Steady turns Fire TV into a patient strength-and-balance coach:

  • Your phone is the camera. Scan a QR code on the TV, prop the phone against a book, and you're tracked. Pose estimation (MediaPipe) runs on the phone. The session screen has four panels: the coach, a mirror of you, your live camera picture with the tracked lines and joints drawn on it, and the counter. A switch on the phone controls the picture: on by default, relayed live and never recorded or stored; off, and only joint positions reach the TV.
  • Coach and mirror. An animated coach demonstrates each move next to a live, mirrored skeleton of you, so "do what I do" works from the sofa.
  • 28 exercises, all tracked. Leg strength, balance, arms and shoulders, mobility, seated cardio and the CDC check, browsable in Fire TV-style rows. Reps are counted and holds timed from the body's own proportions, with one plain-language cue at a time ("Stand all the way up", "Bend your knees a little more"), shown big and spoken aloud.
  • Planned around your health. Setup asks about knee pain, hip replacement, back pain, osteoporosis, shoulder pain, dizziness, a recent fall, a walking aid, and heart or blood pressure. The plan leaves out moves that aren't advised (no twisting or forward bending with osteoporosis, arms below shoulder height with a sore shoulder), swaps in safer ones (hands-assisted standing for knee pain, seated march for dizziness, the Otago balance ladder from feet together to one leg after a fall) and starts gentler. In the library, ruled-out moves are dimmed with the reason.
  • Remote, voice and screen in one flow. The Fire TV remote drives everything (OK pauses, ▶ skips, Back finishes). The phone takes voice commands ("pause", "easier", "next", "I'm OK"). The TV carries the visuals.
  • It checks on you. If your body goes horizontal and stays there, the session stops and a full-screen "Are you OK?" appears. With no answer in 15 seconds, your family contact gets an alert. Saying "I'm OK" sends a follow-up.
  • Clinically grounded progress. It runs the CDC STEADI 30-second chair stand test every two weeks and scores it against age and sex norms. That's the number a doctor actually understands.
  • Set up for your own home, on the TV. First-run setup is built for the remote: name, an age stepper on the D-pad, sex (for the CDC norms), a personal goal and a family contact. Everyone's history is stored on the TV itself, not in the cloud. A ready-made demo household is one click away.
  • It adapts to each person. Each person has a profile and history. The home screen predicts who's in front of the TV from the household's own habits ("Walter usually exercises around 6:30 PM"). Targets progress like a physiotherapist would: more reps after you hit the target twice, fewer after a miss. An AI coach explains today's plan and writes the end-of-session summary ("11 chair stands, up from 8, now in the usual range for your age").
  • Family stays in the loop, with real push notifications. The Family tab shows a QR code. A son or daughter scans it once and subscribes in the free, open-source ntfy app (no account). After that, session summaries and urgent fall alerts arrive as real phone notifications. A Send a test alert button proves it works.
  • It feels like Fire TV. Amazon squid ink and orange, a Prime-blue focus glow, top navigation, and motion everywhere: an animated logo, ambient light, spring focus, rep bursts, ring sweeps, confetti when you hit a target, and pulse rings on the safety alert.

How we built it

  • Exercise library: 28 exercises built from two reusable trackers: one counts reps of any movement described by a body signal and thresholds (with optional calibration to the person's resting pose), the other times holds and scores steadiness. Each exercise is a few lines of signal plus coach keyframes, and the keyframes also drive the demo person, so every exercise is tested automatically.
  • Fire TV app: native Kotlin + Jetpack Compose for TV, the same stack as Amazon's hello-world-fire-tv starter. minSdk 25 so it runs on every Fire TV still getting updates (Fire OS 6+). It has a Leanback launcher banner, full D-pad and media-key handling, and keeps the screen on during workouts. It uses Android TextToSpeech for the coach's voice and an OkHttp WebSocket for live pose data. The household (people, sessions, the family's private notification topic) is saved on the TV as one atomic JSON file. The 10-foot UI follows Fire TV's own look: squid-ink surfaces, orange primary actions, a white outline, spring scale and blue glow on focus, Inter type, and nothing smaller than 20sp. Motion is built from Compose animation primitives (springs, Animatable, Canvas), with shared tokens for durations and easing.
  • Exercise engine: pure Kotlin, no Android dependencies, JVM unit-tested. Every threshold is relative to the person's own body (torso length, thigh length, shoulder width), so it works at any distance from the camera. Sit-to-stand uses knee-below-hip distance over torso length, because knee angles fail when you face the camera. A synthetic demo person, generated from the same keyframes as the coach animation (with jitter, sway and the occasional sloppy rep), drives both the tests and a no-camera demo mode.
  • Phone page: MediaPipe Pose Landmarker (Tasks Vision, WebAssembly + GPU) in the browser, plus the Web Speech API for voice commands, streaming 15 frames a second.
  • The coach's brain: deterministic Otago-style progression rules on the TV decide the plan: more reps after two good sessions, fewer after a miss, a chair stand test every two weeks, and gentler starting targets after 80. An LLM (gpt-oss-120b on Groq, any OpenAI-compatible API works) only explains the plan and writes the end-of-session summary, in warm, plain words with the real numbers. If it is slow or unavailable, the TV writes the note with its own rules, so everything works without it.
  • Companion server: Node + TypeScript + Express + ws, and stateless. It pairs phone and TV by a 6-digit code that expires, relays frames over WebSockets, serves the phone and family pages, and holds the LLM key, so the key never ships in the APK. Requests are rate-limited, validated and size-capped, and nothing about the household is stored. It ships as a Docker image with a Render blueprint (one-click deploy, HTTPS included). For local use, a one-command Cloudflare quick tunnel gives phones the HTTPS they need for camera access.
  • Family notifications: the TV posts straight to ntfy.sh on an unguessable per-household topic, so alerts don't depend on our server. Urgent fall alerts use ntfy's highest priority. The family web page (styled like the Amazon app) shows the same feed live over server-sent events.
  • Built on Windows with the Android TV API 30 emulator, the base of Fire OS 8. We used an AI coding agent (Claude Code) as a pair programmer throughout.

Challenges we ran into

  • No camera on Fire TV. Moving pose estimation to the phone and sending only points solved hardware, privacy and bandwidth in one go. The hard part was making the analysis robust to whatever angle and distance a propped-up phone ends up at.
  • Knee angles lie from the front. When you sit facing the camera, your thighs point at the lens. We switched to body-relative distances, and validated every tracker against a synthetic person before touching a camera.
  • Avoiding false fall alarms. Bending to sit down must not trigger an alert, so the check needs the torso near horizontal (or the head below the hips) for a sustained 1.5 seconds. It's tested against full sessions of normal movement.
  • Ten-foot UI for older eyes. We had to fit plans, progress and controls on a 960×540 dp canvas with nothing under 20sp, and keep focus legible from a sofa.
  • Typing on a TV. Compose TextFields swallow D-pad Up/Down, so the remote got trapped inside the setup form. We added explicit focus moves, and replaced typing with steppers and chips wherever we could.
  • Keeping the AI honest. Early summaries invented chair-stand scores and called the family contact the wrong relation. Now the model only gets numbers that exist (chair-stand data only on test days, the family contact's name and relation passed explicitly), its JSON output is validated, and the rules text is the fallback.
  • Fire TV development on Windows. The Vega SDK is macOS/Ubuntu only, so we built for Fire OS with the Android TV emulator (details in the feedback below).

Accomplishments that we're proud of

  • A complete, installable product, not a mock-up: first-run setup, household home screen, exercise library, progress charts, pairing, coached session, safety flow, summary, family notifications and settings, all driven by a TV remote. Released as an APK.
  • The safety flow. Simulate a fall and, 15 seconds later, a real push notification lands on the family's phone with a clear, actionable message.
  • A validated clinical screening test (CDC STEADI chair stand) built in, with an honest framing: "share this with your doctor."
  • 34 automated tests (28 for the exercises, health planning and coach rules, 6 for the server), plus a demo person, so anyone can try it without a camera.
  • Tested end to end with real footage: a real video of a person exercising, fed through the real phone page (MediaPipe), the hosted companion and the TV, with the live AI coach and real ntfy pushes. Those runs found and fixed four bugs (a fall alert that re-fired while the person was getting up, squats counted as heel raises and as march steps, and the AI quoting raw scores); the engine fixes are covered by new tests.
  • Released the engine as a standalone open-source library, steady-pose.

What we learned

  • The best TV interaction for older adults is mostly not interaction. Countdowns that start on their own, one cue at a time, OK to pause, and a coach that talks.
  • Body-relative geometry beats absolute angles for real-world pose coaching.
  • AI is most useful at the edges. Rules decide what's safe to change; the language model explains it like a kind physiotherapist.

What's next

  • Pilot with a physiotherapy clinic: clinician-assigned plans and a weekly report for the clinician.
  • Alexa: "Alexa, start my balance exercises" through Fire TV's voice remote, and a daily nudge on Echo devices.
  • Fire TV Cube with a USB webcam as an alternative to the phone.
  • More of the Otago programme (tandem stance, walking patterns), and gentle progressions toward stepping without support.

Built With

Share this project:

Updates

Submission history