Trace

Inspiration

Everyone loses things: keys, glasses, a phone charger. For most of us it's a daily annoyance we've accepted as "just the way things work." But for people with sight loss or dementia, losing track of belongings means lost independence, stress and reliance on others to find everyday items.

We asked a simple question: search engines made the digital world searchable, so why can't we search our physical one? Modern vision-language models can now name almost any object and describe where it is, which made this possible for the first time.

What it does

Trace builds a searchable memory of your home from your phone's camera.

Scan: stream or record a room from one or more phones.
Understand: AI identifies every object and describes where it is in plain English, relative to furniture: "keys, on the kitchen counter, next to the kettle."
Remember: each scan is timestamped, so the app tracks when things move or go missing.
Ask: type or say "Where are my keys?" and get an answer with the photo it was last seen in: "Last seen on the kitchen counter at 14:32."

Showing the photo matters: carers and sighted users can check the answer at a glance, and the app tells you when it isn't sure rather than guessing.

How we built it

Capture: a Python multi-camera script (multicam.py) that pulls live video from several phones over the network and samples keyframes, so we don't send near-identical frames to the model.
Object understanding: a vision-language model [Gemini] reads each keyframe and returns structured JSON: object name, the fixed furniture it's on or near, a plain-English location, and a confidence score.
Cross-checking: [YOLO-World], an open-vocabulary detector, looks for the objects the language model named. When both models agree, the object is marked verified; when only the language model sees it, the app says "I think…" instead of stating it as fact.
Memory: a [SQLite] database organised as Room → Landmarks (furniture that doesn't move) → Objects, with timestamps, so we can compare scans and find what moved.
Asking questions: the whole home inventory fits in an LLM's context, so questions are answered directly over it, with no retrieval pipeline needed. [Voice in and out uses the browser's Web Speech API.]
Front end: [describe your UI: web app, framework]

Challenges we ran into

Duplicate objects: the same mug appears in dozens of frames. We had to merge sightings into a single object, keeping names consistent across frames.
Hallucinations: language models sometimes confidently "see" objects that aren't there, often whatever usually goes with the scene, like a remote next to a sofa. For a blind user, a wrong answer is worse than no answer, which is why we added cross-checking and confidence levels.
Networking: streaming from several phones on venue Wi-Fi was harder than expected. Devices on the same network often can't reach each other, so [we switched to a phone hotspot / describe your fix].
Small and cluttered items: keys under papers are hard. [Describe anything you did: higher-resolution frames, cropping into tiles.]

Accomplishments that we're proud of

A working pipeline from live phone cameras to plain-English answers about where things are.
Combining two independent models to catch hallucinations, so the app is honest about what it doesn't know.
[Add your favourite demo moment.]

What we learned

Describing where things are relative to furniture is far more useful, and easier to build, than 3D reconstruction or raw coordinates.
Language models and dedicated object detectors are good at different things. The language model is better at naming objects and describing their surroundings, the detector at precise positions and confidence, and they work best together.
Responsible AI isn't an add-on. For vulnerable users, saying how confident the app is turned out to be one of the core features.

Responsible AI & privacy

Video of someone's home is deeply personal. [Name] stores the object inventory and a small evidence photo for each item, not the full footage. The app is designed to support people with dementia and their carers, not to replace care, and it always says when it is uncertain.