Patch Hours

A Fire TV app for kids patching a lazy eye: their show plays only while the phone camera sees the patch, and pauses the moment it comes off. Parents get real minutes worn.

Comment

Inspiration

About half of the eye-patching prescribed to children with amblyopia is never worn. In 152 children whose patching was measured by an occlusion dose monitor, mean compliance was 44%, and on 42% of days no patch was worn at all (Wallace et al., IOVS 2013). Families keep a diary from memory. Meanwhile the child is usually watching TV.

What it does

Patch Hours is a Fire TV app built for Vega OS and a page on the parent's phone. The TV shows a pairing code; the phone scans it and turns on its camera. The parent teaches the phone this child's patch in a few seconds (patch on, no patch, a hand over the eye) and sees held-out precision and recall. While the phone sees the patch, the cartoon plays on the TV and the minutes count. When the patch comes off, the TV pauses within about a second and a half with "Patch back on to keep watching"; the web version of the TV also says it aloud, because a four-year-old can't read. A hand over the eye is refused.

The parent gets a week sheet: minutes the camera saw, kept apart from minutes they add by hand. Once a week they tap "Write this week's note". Plain code totals the minutes; Amazon Nova phrases them as a short note for the child's orthoptist and a one-line cheer for the child. Every number in the note is checked against the table (if one doesn't match, a plain template is used), and a Bedrock guardrail screens both texts for unsafe content and treatment advice. Amazon Polly speaks the cheer, and the web TV plays it the next time the patch goes on.

How we built it

  • Fire TV app: React Native for Vega (Vega SDK 0.24) and the @amazon-devices/react-native-w3cmedia Video component, run and recorded on the Vega Virtual Device. It pairs, polls the session every second, and plays or pauses only on a change of state; if the API is unreachable for four seconds it pauses. The cartoon plays with sound; if the player's clock stops (the virtual device's audio stalls under WSL2), the app switches to a silent copy at the same point and says "Sound off".
  • Phone check: MediaPipe Face Landmarker crops the prescribed eye; MediaPipe Image Embedder turns the crop into a vector; a per-child nearest-neighbour vote decides patch / no patch / hand. It all runs in the browser: only on/off, a class, a confidence and timestamps leave the phone, and the API rejects anything else.
  • API: Next.js route handlers on Vercel with Neon Postgres; versioned writes so concurrent heartbeats never lose minutes; minutes counted only between "on" heartbeats under five seconds apart.
  • AWS: Amazon Nova 2 Lite, served by Amazon Bedrock, writes the week note; Bedrock Guardrails (content filters plus a denied "treatment changes" topic) screens every generated text; Amazon Polly (generative voice) speaks the cheer and the pause prompt; Amazon S3 (private, 30-day expiry) keeps the clips. One CloudFormation template (aws/patch-hours.yaml) creates the guardrail, bucket, OIDC provider and IAM role; the app on Vercel swaps its own OIDC token for that role, so no AWS keys are stored anywhere. Our account's Bedrock model quota was 0 and the increase was declined, so the app reaches Nova through Vercel AI Gateway, which serves it from Bedrock; with model access, one setting switches it to the Bedrock Converse API directly.
  • Ready to run: a release zip with run.sh that checks the machine, starts the virtual device, installs and opens the app, and can stand in for the phone.

Challenges we ran into

The virtual device under WSL2 needed libpulse0 and the kvm group, and after a WSL restart it died at boot because its runtime folder was gone; each time the CLI said only "Deadline reached". The video played but stayed invisible until we learned it renders under the UI (views over it must be transparent). Its audio clock stalled and froze playback, so the app now watches the clock and falls back to a silent copy. A spoken prompt through a second media player froze the app, which we found only in the device's crash history. On AWS, the Bedrock model quota on our new account was 0 and the increase was declined. All 14 Vega issues are in FRICTION_LOG.md.

Accomplishments that we're proud of

One continuous recorded run, phone to TV: pairing, calibration, play, a pause 1.5 s after the patch came off, the hand refused, and the show resuming. A zip that installs and runs the TV app with one command and can stand in for the phone. A week note whose every number is checked before a parent or clinician sees it. And vvd-kit, an open-source tool that turns our friction log into a 3-second check other Vega builders can run.

What we learned

On Vega, video lives on its own plane, and the emulator console (screenrecord) is the reliable way to capture it. When a model writes for a clinic, let plain code own the numbers and make the model's job phrasing only.

What's next

Testing with real families and orthoptists, a physical Fire TV, clinic accounts, and pairing that survives a TV restart on its own.

What's real and what's simulated

Real: the Vega app, the phone check, the API and the week note, all live. Simulated in the video: the face is synthetic (AI-generated, not a child), the TV is the Vega Virtual Device, and the clip plays without sound there. The week note and the spoken prompts were added after the video was recorded. Big Buck Bunny is © Blender Foundation, CC BY 3.0.

Built With

Share this project:

Updates

Submission history