Recall

55 million people worldwide live with dementia, most cared for by family. Recall gives the home a memory: it sees what happens, remembers it, and speaks up when it matters, so you can worry less.

Comment

Inspiration

My grandfather had dementia. One afternoon he walked out of the house on his own, and our family spent hours searching for him. On ordinary days it was smaller things: his glasses, the TV remote, and never being sure whether he'd already taken his medicine. Nobody in the family ever knew for certain. We just worried.

That's the daily reality for millions of families. 55 million people worldwide live with dementia, a number expected to reach 139 million by 2050, and most of them are cared for at home by family. Dementia doesn't erase everything at once. Day to day, it erases the last few minutes: Did I already take this? Where did I just put that? Is someone still home?

I wanted to build something that holds onto those minutes, quietly, at home, for the person living with dementia and for the people who love them.

What it does

Recall gives the home a memory. A camera watches the spaces that matter, the kitchen counter and the front door, and Recall understands what happens there in real time.

Medication safety

  • Recognizes when the medication is picked up, and whether a hand goes to the mouth afterward. That's logged as "dose appears taken," with a short video clip.
  • If the medication is picked up again within the dose window, Recall speaks up immediately, before a second dose: a chime, then a familiar family member's voice (generated with ElevenLabs, played offline): "You may have already taken this one. Please check with me first."
  • At the same moment, the caregiver's phone gets an alert with the clip, plus one-tap "Call" and "Mark as handled."
  • It never tells anyone to take or skip a dose. When it can't see clearly, it logs "Not sure" and asks the family to check instead of guessing.

Memory for everyday things

  • Ask "Where are my glasses?" by voice or with one tap, and Recall answers out loud: "I last saw them going into the basket at 2:14 PM," with the clip as proof.
  • It understands items it can no longer see: things dropped into a basket, or glasses that left the room while being worn ("You were wearing them").
  • New items are added by typing their name. There's no training.

Leaving the house

  • A second camera watches the doorway. When someone walks out, the caregiver gets "Left the house at 9:42 PM" with the clip, the moment it happens, and a gentle spoken reminder plays at home.

For the caregiver

  • A phone app shows today's doses against the schedule (observed / not sure / no observation), alerts, where the person's things are, and a plain-English activity timeline, each with video proof.
  • "Talk to Mom" turns the spoken reminders on or off; alerts still reach the caregiver either way.

Private by design

  • Everything runs offline, on a computer inside the home. Video never goes to the cloud.

How I built it

Perception (three models fused in real time)

  • YOLO-World (large, v2), an open-vocabulary detector, finds the person's items from plain-English descriptions like "pill bottle" or "fish oil bottle," with no custom training. It runs on Apple Silicon through PyTorch MPS, with a household vocabulary that includes look-alike objects (sprays, mugs, phones) so they don't get mistaken for tracked items.
  • MediaPipe Hands for 21-point hand landmarks: what's being held, and by which hand.
  • MediaPipe Pose for wrists, fingertips, the mouth and shoulder width. Hand-to-mouth distance is measured relative to shoulder width, so it works at any distance from the camera.

Scene understanding

  • Recall finds the counter and the basket on its own, from detections, and smooths and tracks them over time. If the camera gets bumped, the zones move with it. There's no setup or calibration.
  • Location uses where an object touches the surface (bottom-center of its box), so something in front of the basket isn't mistaken for something in it.

Memory: a per-item state machine

  • Every item moves through states: on a surface, in hand, worn, last seen going into a zone, last seen being worn.
  • Noisy detections become stable facts through debouncing, hysteresis (it's harder to lose a tracked item than to start tracking one), and one rule: objects don't move by themselves. If no hand touched an item, it's still there, even if the camera briefly lost it.
  • Every change is written to a SQLite event log, the single source of truth for answers, alerts and the timeline.

Medication logic (built to avoid false alarms)

  • A pickup only counts once it's confirmed: the bottle actually moved away from its resting spot, or a hand covered it and it stayed gone. Sitting in front of it or briefly blocking the view never counts.
  • Medication must look like a bottle (shape check) and must not overlap a wallet, phone or look-alike detection.
  • "Dose appears taken" needs the full sequence: confirmed pickup, then a hand at the mouth for a sustained moment, with the mouth visible.
  • Unclear attempts still count for safety, so a second reach after an unclear one also triggers the reminder.

Door camera

  • The Mac's webcam runs a separate YOLOv8n person detector on the CPU, in its own thread, with a centroid tracker and a virtual door line, so the kitchen camera never slows down.

Voice

  • "Tap to ask" records in the browser, stops on silence, and is transcribed on-device with faster-whisper (base.en). Simple keyword intents map to answers, which are spoken back out loud. No LLM decides anything.

Proof

  • A rolling 10-second video buffer saves a clip around every event (6 s before, 2 s after), encoded to H.264 in a background thread so the camera never stutters.

Server and apps

  • FastAPI streams the live annotated video (MJPEG) and pushes every event and a state snapshot over WebSockets.
  • The kitchen hub and the caregiver phone app are a React + TypeScript app (Vite, Tailwind v4, Motion, Vaul, Radix), built on Apple's iOS system colors, type scale and spacing, with light and dark mode, spring animations and an iOS-style liquid selection bubble.
  • A "Judge view" shows the pipeline live: models running, FPS, each item's state, the dose state machine and the live event log. A "How it works" page explains every part.

Challenges

  • False alarms were the biggest risk. An early version mistook a black wallet for medication and sent a "possible second dose" alert. For a safety product, that's unacceptable. I added shape checks, look-alike filtering, higher thresholds and the confirmed-pickup rule, because a false alarm erodes trust faster than anything.
  • Things disappear all the time. A hand covers the bottle, the wallet drops behind the basket wall, a person walks in front of the counter. Teaching Recall to tell "hidden" from "moved" was the hardest problem.
  • Hand-to-mouth at real distances. Face landmarks failed once the person was a meter or two away, so I rebuilt it on full-body pose and measured everything relative to body size.
  • Running everything on one laptop. Three vision models, a door camera and speech recognition compete for the same CPU and GPU. Two threads on Apple's GPU crashed the app, so the door detector runs on the CPU and the kitchen keeps the GPU.
  • Real-world hardware. Camera lenses switching mid-session, Continuity Camera dropping on low battery, window glare, and getting one stable camera angle in a real room.

Accomplishments

  • The full loop works live and end to end: pick up the medication twice, Recall speaks up before the second dose, and the caregiver's phone gets the alert with the clip within seconds.
  • Door alerts, voice questions and video proof, all running offline on one laptop.
  • No training, no setup: items from plain English, zones found automatically.
  • It reasons about what it can't see, and says "not sure" when it should.

What I learned

  • In health, honest uncertainty is a feature. "Not sure, please check" builds more trust than a confident wrong answer.
  • The hard part of computer vision isn't detecting objects. It's turning flickering detections into decisions you'd trust with someone's safety.
  • The best interface for someone with dementia is almost no interface: just ask, and it answers.

What's next

  • Recognizing the person's own items (their glasses vs. anyone's) from a few photos
  • Multiple rooms writing to one shared home memory
  • Medication timing support for Parkinson's, and memory support after stroke or brain injury
  • Running on a small dedicated home device instead of a laptop
  • A pilot with real families to measure how many check-in calls and searches it replaces

Built With

Share this project:

Updates

Submission history