Mantel: the living-room TV, in your family's words

A Fire TV app for a person living with dementia. It answers the question asked forty times a day in the family's own words and voice, says who is at the door, and flags sudden change.

Comment

Mantel turns the living-room TV into a patient memory for a person living with dementia. It answers the questions they ask all day in their family's own words and voice, tells them who is at the door, and alerts the family when something changes. It runs on Fire OS and on Vega OS, and every word the TV says was written or approved by the family.

I have seen how dementia makes a person ask the same questions again and again. Mantel puts the family's answers on the screen that person already faces, in their own words, every time.

Four Mantel screens on the Vega Virtual Device The Mantel Vega app on the Vega Virtual Device: the Today screen, the family's answer to "Where is Robert?", an unexpected caller at the Ring doorbell, and the night screen.

Why it matters

  • 7.4 million Americans aged 65 and older live with Alzheimer's dementia (Alzheimer's Association, 2026 Facts and Figures).
  • The same questions come all day. What day is it? When is Sarah coming? Where is Robert? Families answer with whiteboards, day clocks and patience, and by answering again.
  • The TV is the screen they already face. It is on for hours, it is large, and nobody has to learn it.
  • A sudden change in confusion can be delirium, often from a treatable infection, and families see it late because every day already looks hard.

What it does

  • Today. When the webcam sees someone in the room, the screen shows the day, the part of the day, the date and what happens next ("Sarah is coming at 4 o'clock", with her photo). It answers the common questions before they are asked.
  • Kind Answers. Asked out loud, with no wake word, the TV answers with the words a family member wrote, in their recorded voice, the same way the fortieth time as the first.
  • At the Door. When the Ring doorbell rings, the TV says whether a visit or a delivery was planned. When nothing was: "You're not expecting anyone. You don't need to open the door. Sarah can see who it is." Sarah gets the doorbell's snapshot. Mantel never identifies a caller.
  • Moments, evening and night. Family photos and stories in family voices, a calmer evening, and a dim night screen. An outside door opening at night alerts the family.
  • The Change Signal. Each morning, Mantel compares yesterday with the person's own last four weeks and tells the family about a sudden change: "Consider calling Margaret's doctor." It never diagnoses.

The answers grow from what is actually asked

The answer loop A question with no answer gets "Let's look at today together." and goes to Sarah's phone with its count. She writes the answer once, and the TV gives it from then on.

The hardest questions get the most care. Margaret's husband Robert died two years ago. For "Where is Robert?", Sarah chooses topic by topic whether the TV tells, redirects or comforts, starting from published caregiving guidance, and records it in her own voice. Mantel never makes that choice, and nothing reaches the TV until the primary caregiver approves it.

The family app Sarah's phone: the questions that have no answer yet, the answer editor with tell, redirect and comfort, and a Change Signal alert with the counts behind it.

How it works

How Mantel works Camera frames and audio stay on the TV. The server receives events, never images or sound from the room.

  • One TypeScript core holds the rules (Today, the answer matcher, the door decision, the Change Signal, the digest). It runs on the TV, on the server and in the tests, so the Today screen and the family's answers keep working with no network.
  • A deterministic matcher. A question that names a person can only be answered by a topic about that person. A question naming nobody never gets a topic about somebody. "Where is he?" with no name gets no answer. A wrong answer is the failure that matters, so the matcher is built to say "Let's look at today together." instead.
  • The Ring doorbell. Ring's signed webhooks (HMAC over the raw body, idempotent on request_id) decide the door card before the reply, and the frame at the press arrives through Ring's image download. A newer card replaces an older one, and a card the server no longer holds closes.
  • The Change Signal reads four series: questions asked, night-time activity, night door openings and waking time. Each is compared with the person's own 28-day baseline, so a household that always asks a lot never raises an alert for that.
  • Language models only where a person approves the result. Amazon Bedrock drafts photo captions and calmer wordings of answers; each draft passes checks (no new names, no new numbers, no death words) and waits for the family. The evening digest is a template over the day's counts, because a caregiver reads it to decide whether to call a doctor.

Built for Fire TV

  • Fire OS. A Kotlin app: a full-screen WebView runs the React TV interface from the APK's own assets, with the device's senses behind a JavaScript bridge. CameraX reads a USB webcam on the Fire TV Cube and Android's face detector turns each frame into a count of faces. Vosk's small English model recognises speech on the device. The remote's OK reads the day, and Back never leaves the app.
  • Vega OS. A React Native for Vega app with the same screens, built natively, importing the same core through Metro. Vega's media player opens https only, so the app fetches each family recording itself, wraps its MP3 frames in an MP4 in JavaScript (losslessly) and plays it through Media Source Extensions.
  • Care mode. An Android library opens Mantel at boot and brings it back when the TV is left idle on another app, so the screen a person returns to is always theirs.
  • A 10-foot interface for older eyes. Atkinson Hyperlegible, digits with a plain zero, one message per screen, and sentences a person can read from across the room.

Results

Change Signal against alerting on questions alone 40 held-out simulated households, each with a matched null. Mantel catches more strong changes than alerting on questions alone, with five times fewer false alerts.

Measured results Every figure comes from a command in the repository: pnpm eval, pnpm bench, pnpm test.
questions answered with another topic's answer 0, across 165 held-out sentences and 1,056 spoken ones pnpm eval, docs/EVAL.md
first held-out read, written before it was scored 53 of 57 answered right; 2 of 35 unanswerable sentences got a true statement from the clock or plan docs/eval/holdout_read_1.txt
second held-out read 36 of 43 answered right; 2 of 30 got a true statement docs/eval/holdout2_read_1.txt
spoken: every question said by four voices, recognised by the TV's own Vosk model 81.4 % to 89.2 % right per set, 0 wrong topics pnpm eval
Change Signal, strong sudden changes caught within 48 hours 70.0 % at 0.148 false alerts per household-month (questions alone: 52.5 % at 0.780) docs/eval/signal_holdout_read_1.txt
speech end to end in the Android TV API 30 emulator (the Android base of Fire OS 8), the app recognised "when is sarah coming" from audio on the device and showed Sarah's answer docs/FIRE_TV.md
Vega OS Today, answers, Sarah's recording through MSE, both door cards, a family message, a story from the remote and the night screen on the Vega Virtual Device docs/VEGA.md, the demo video
Amazon Bedrock drafts, openai.gpt-oss-120b 1.3 to 4.1 s per draft pnpm check
Amazon Polly, neural 410 ms for a sentence, cached once per line pnpm check
tests 108 TypeScript (core, server over HTTP, Ring, CDK, the Vega MP4 wrapper, the real interfaces in Chromium), 5 on the DynamoDB engine, 8 Android unit tests, 1 Vega app test pnpm verify, GitHub Actions

The Change Signal figures come from a simulation of the method and make no clinical claim. docs/EVAL.md gives the commit order, the method and every number.

Try it

Live on AWS: https://wcnre0e2hh.execute-api.us-east-1.amazonaws.com/tv/?token=demo-tv&dev=1 is the TV in a browser, and https://wcnre0e2hh.execute-api.us-east-1.amazonaws.com/family/ is Sarah's phone (sign in as Sarah). On the TV, the dev panel stands in for the camera and microphone: type "where is robert" and the TV answers with Sarah's recording. The demo household resets each day.

On a Fire TV with Fire OS: docs/FIRE_TV.md. On the Vega Virtual Device: docs/VEGA.md. Locally, with no account of any kind: pnpm install && pnpm dev.

Built With

Share this project:

Updates

Submission history