Keepsake

keepsake

Inspiration

Half of our team has a grandparent living with dementia, so this idea came from something we've experienced in our own families. We've heard questions like “Have you seen my pills?” or “Where are my glasses?” repeated throughout the day. We wanted to build something that could help with those moments, both for the person looking for their things and for the family members helping them.

More than 7 million Americans live with Alzheimer's disease, and most still live at home. Misplacing everyday items can be frustrating and make it harder to feel independent. Sometimes, a person may believe someone has taken their belongings. That made us interested in using photos, so there would be something concrete to refer to when helping them look.

Existing tools don't fit this problem:

Tracker tags only work on things you remembered to tag.
Smart glasses ask someone with memory loss to remember to wear, charge, and keep track of a device, so the glasses become one more lost item.

That led us to Keepsake: a camera-based memory assistant that remembers where everyday items were last seen and saves a photo. We chose a problem we knew from our own families and built something we hoped could make that part of their day a little easier.

What It Does

Keepsake is a camera-based memory assistant that helps people living with dementia keep track of everyday belongings.

Using a phone camera, it detects items like glasses, keys, and wallets, remembers when they were last seen, and saves a photo. Users can ask where something is and receive a spoken answer alongside the photo, with the item highlighted. The idea is to give them something they can recognize and use to retrace their steps.

In a fixed room setup, Keepsake can scan furniture and use familiar locations like “near the couch” to describe where an item was seen. The phone can also be worn on a chest mount to capture items as the person moves around. In that mode, it refers to the last sighting and photo rather than treating the camera’s position as a location in the room.

Of course, a camera can't see everything. An item might be covered, taken out of view, or moved after the last photo. Keepsake reports what it last saw and makes that uncertainty clear. It doesn't need tags attached to belongings, and video is processed locally on the computer.

How We Built It

Camera. An iPhone streams to a Mac over Continuity Camera, selected by name so the camera index can't shift between runs. OpenCV reads the frames.
Detection. Ultralytics YOLO-World (yolov8s-worldv2), an open-vocabulary detector we prompt with each item's name. This finds things standard detectors can't, like keys, wallets, AirPods, and glasses. It runs at 1280 px on Apple MPS at about 5 fps, with ByteTrack for tracking and a per-item confidence threshold.
Room areas. Scan room runs the larger YOLO11m model over a few still frames, keeps furniture it sees reliably, merges duplicates, and numbers repeats ("Chair 1," "Chair 2"). Each item gets the most specific area it sits in, "near the …" if it's just outside one, or the left, middle, or right of the room if nothing fits.
Wearable mode. MediaPipe hand landmarks plus OpenAI CLIP embeddings. When a hand touches an item for half a second and lets go, the item is tracked. A per-item tracker follows it as held, put down, out of view, or with you, and saves the sharpest recent frame as the last-seen photo.
Voice. Browser speech recognition for input; answers spoken in a user-selectable Grok voice (xAI text-to-speech), with the browser voice as a fallback. The xAI key stays on the server.
App. FastAPI and SQLite on the backend; React, Vite, and Tailwind on the frontend, with a List view, a Map view drawn over the scanned room, item detail with photo history, and a privacy panel.
Deterministic core. Where an item comes only from the database and the state machine, never from a language model. The model just turns facts into a sentence, so it can't invent a place.

Challenges We Ran Into

Our biggest challenge was switching from a stationary camera to a wearable one. We initially designed Keepsake to watch a single room, but realized that once someone left that room, it could no longer help track their belongings. Installing cameras in every room would also make the setup expensive and inconvenient. Using a phone on a wearable mount made it more accessible and allowed it to follow the person throughout their day.

That decision meant rethinking much of the design. With a stationary camera, we could use furniture and fixed areas to describe where an item was. With a wearable camera, the view changes constantly, and an object disappearing from the frame might just mean the person turned around. We had to change how we tracked items, recorded memories, and decided when to give reminders, using hand interactions and the last recorded sighting instead of relying on a stable view of the room.

Moving the camera introduced another problem: everything appeared to move at once. For a fixed camera, we added a scan process that rechecks the room. Wearable mode needed a different approach, using sightings and hand interactions rather than fixed room locations.

Accomplishments That We're Proud Of

It doesn't lie. Moved mid-carry, camera bumped, never seen: each case gets a truthful answer instead of a confident guess. For a user who can't check the answer, that matters more than anything else.
The photo is the answer. A photo of the item in the user's own room, taken the moment it was put down, ends the search and ends the argument about whether someone took it.
Nothing to wear, tag, or charge. One phone on a shelf covers a whole room. Wearable mode is there when the person wants it, never required.
Everything stays home. Detection, tracking, and memory all run on one laptop. Photos stay on disk under the user's control, with a switch to pause photo saving and a button to forget everything.
Twelve items, zero training, thanks to open-vocabulary detection and per-item tuning.

What We Learned

The hard part isn't detecting objects. It's deciding when something actually happened. A detector sees a bottle in every frame. Deciding "the bottle moved to the desk" took a state machine, motion checks, and a lot of tuning to reject the dozen ways a frame can mislead you.
Don't let a language model decide where something is. It will happily name a place. We kept location in code and the database, and used the model only for phrasing.
Designing for memory loss changes the priorities. "I don't know" is a good answer. A photo beats a sentence. And the best interface is one the person never has to remember to use.

What's Next for Keepsake

Caregivers in the loop. Text alerts and text-back ("Where are Mom's glasses?") so family can help from afar, plus a warning when a critical item like an inhaler or pill organizer is left behind as the person heads out the door.
Medication check-ins. "Did I take my pills today?" answered from when the pill organizer was last opened.
More rooms. Several cameras sharing one memory, so items can be followed from room to room.
Testing with real users. Assumptions are risky when designing for someone else's mind. Next, we want to test Keepsake with people living with memory loss and their caregivers, and let their feedback set the priorities.