Inspiration

Several published devices already whisper a visitor's name to someone living with dementia. None of them asks whether it should. Correcting someone in later-stage dementia can cause distress, and the worst case is a device cheerfully naming a husband who died last year. We wanted to build the missing piece: a face-recall aid where the decision to speak is a careful judgement, controlled by the caregiver, rather than an automatic reaction to every face.

What it does

When someone the caregiver has enrolled approaches, a quiet voice in the wearer's earbuds says, for example, "Sarah, your daughter. She brought the roses." The same words appear as a large caption and in a daily transcript. A policy guard decides first. It stays silent for people flagged as sensitive, recent repeats, uncertain matches and paused sessions, and switches to a gentle warmth cue in later stages or at sundowning time.

How we built it

It's a React and Vite web app on the phone. Faces are matched on the phone with face-api, and face templates stay in IndexedDB, never uploaded. The policy guard is a pure, tested TypeScript function. NVIDIA Nemotron 3.5 Lightning on Nebius Token Factory writes each spoken line from text only, behind a validator with a template fallback. Nemotron 3 Ultra writes a nightly caregiver digest, packaged as a Nebius Serverless Job. The Express server accepts text only.

Challenges we ran into

Keeping safety decisions out of the model was the main design challenge: Nemotron writes the words, while deterministic code decides whether to speak. The first face-model run took about 11 seconds, so we warm up at start and write lines ahead of time. Phones require HTTPS for the camera and Android blocks unprompted speech. We fixed a bug where captions could appear before their transcript was saved. Meta's policy on face identification kept us on the phone camera.

Accomplishments that we're proud of

The rule against naming a deceased spouse is enforced in code and tested at every stage, not left to a prompt. Face data never leaves the phone, and the server drops any field it doesn't recognise. Every enrolment needs a consent record, and removing someone deletes their photos too. The live path needs no network. The two Nemotron models are matched to their jobs: one fast and frequent, one for deeper reasoning.

What we learned

Recognising faces is a commodity; deciding whether to speak is the real product. Privacy is far easier to guarantee when it's built into the architecture than when it's a promise in a policy. Matching the model to the task, Lightning for quick wording and Ultra for summaries, kept the app responsive and our credits under control. Most of all, we learned how much we don't know about dementia care.

What's next for Face recall

First, review the two-stage model, the wording rules and the sensitive-relation handling with dementia care professionals and caregivers. Then seek Meta's accessibility approval to move from the phone camera to camera glasses, using the integration we've already written. We'd also like medication reminders, gentle wayfinding home and richer caregiver controls, each with the same rule that silence comes first.