Inspiration
A neighbor's family went through the moment when a woman with dementia didn't recognize her own husband and son. The hard part wasn't a failed lookup. It was the connection that got interrupted in that moment, for her and for them.
People in the early to middle stages of dementia often still live at home and still wear glasses. What they lose is small things at the worst times: a name, a relationship, whether they took their morning pills. We wanted to put what the family already knows into her ear at the moment it helps. We didn't want to try to give her a perfect memory.
What it does
Myst is an Android app paired to Ray-Ban Meta glasses. The wearer hears one calm voice and never configures anything. Her caregiver sets everything up on the phone.
Familiar Faces. She asks "Who is this?" The glasses take one photo, Muse Spark compares it with the family members who consented to be enrolled, and she hears a short whisper: name, relationship, and one memory the family wrote. For example, "Daniel, your son." If it isn't sure, it says so. It never guesses a name, never invents a memory, and never quizzes her.
Medication Routine. The caregiver enters each medication, its bottle photos, the schedule, and whether it's taken with food. At dose time, Myst walks her through the bottle check. It compares what she's holding to the caregiver's record using the photo and the label text, and it only names a bottle when both agree. A dose is logged as "reported taken," never as "taken," and a missed or wrong dose alerts the caregiver. Myst never advises on dosing.
Consent is built in. The caregiver records it per person, and enrollment can be edited or revoked, which deletes the person's record and photo. Nothing is kept: no frames, faces or audio. The memory content stays on the phone.
What is live and what is not
Live on real glasses: photo capture, audio to the glasses, voice trigger (experimental), Muse face matching, and the enrollment tabs. Still a mock or in progress: [the bottle check has only run on an emulator so far; the full dose flow, re-reminders and caregiver alerts are in progress]. Roadmap: daily summary to remote family, pillbox check, and passive approach alerts. How we built it The whole stack is Meta's. There is no third-party recognition service, model or backend.
Ray-Ban Meta glasses and the Wearables Device Access Toolkit for the camera and the speech module. Muse Spark called over plain HTTPS from the phone, for both face matching and bottle label reading. Android as the host OS, since the toolkit requires it. Each request opens one short session, takes one photo, and closes. That fits our consent rule and saves battery. The phone sends a small JPEG to Muse, then a response guard checks the answer before it's spoken. The guard blocks answers over 20 words, banned phrases like "remember" and "you forgot," questions, and any name that isn't from the family-written record. Android's text to speech then plays the answer through the glasses over Bluetooth. Every demo element is labeled live, mock or roadmap in the code and on screen.
Challenges we ran into
The platform's limits. There's no documented glasses button for apps, no Muse text to speech, and no playback call in the toolkit. We used a spoken trigger through the toolkit's speech module, labeled experimental, and Android's built-in speech for output. Flaky connections. The glasses sometimes refused capture with "update required" or "device unavailable," and recovered after reconnecting. One recording came out silent and the next had sound. We haven't explained either one. Muse spends its token budget on hidden reasoning. With default settings, a simple image question often came back empty. Setting reasoning_effort to minimal brought one test from 2 to 3 s down to under 1 s. For face matching, low worked better than minimal, which returned no name. Facial identification is restricted by Meta's policy. We asked a Meta organizer, who said a consent form in the app is enough for a hackathon. For a real product, the approval process would be much more involved. We explain that path in our pitch and don't build it. No OCR fallback. With a Meta-only stack, the bottle check has to rely on Muse alone, so we made two independent checks agree before it names anything. Accomplishments that we're proud of A full loop that runs on real glasses and a real phone: glasses photo, then Muse, then a whisper back through the glasses. In a small early sample, Muse's face-match verdict was "match" 5 of 5 times with confidence 0.95 to 0.99, in 1.5 to 2.5 s per call. That is Muse's verdict, not independently confirmed accuracy. A product that refuses to guess. Uncertain means it says "not sure," and names and memories come only from what the family entered. Consent, revocation and no-retention built in from the start, not added at the end. An honest demo. We say what's live, what's mock, and what's roadmap. [Measured numbers once you have them: bottle check right vs wrong out of 10, wrong-name count across 20 trials, trigger-to-speech time.]
What we learned
Presence matters more than data. The family already knows the schedule, the bottles and the people. Myst's job is to move that knowledge into her ear at the right moment. On a vulnerable user's device, what the system refuses to say is as important as what it says. Hardware surprises you. We spent real time on reconnects and permissions that no document had warned us about. On the Meta stack, the real limits are policy and platform support, not model quality. What's next for Myst Finish and rehearse the medication flow on real glasses: the bottle check, the double-dose block, re-reminders at +10 and +20 minutes, and caregiver alerts through the WhatsApp Cloud API. Measure and publish accuracy: false "yes" on bottles, wrong-name rate, and latency. For real-world use: Meta's approval for identification, a fuller consent and revocation process, and a decision on whether frames can safely go to a cloud model or the matching moves onto the phone. Caregiver tools: a shared timeline of visits and doses, and a daily summary for remote family. Wider testing with caregivers and people living with early-stage memory loss, so we can find out whether the voice feels respectful.
Log in or sign up for Devpost to join the conversation.