-
-
Remi: Glasses that whisper who's here, so nobody with dementia feels like a stranger
Inspiration
"Who is this?" is the question families dread. More than 7 million Americans over 65 are living with Alzheimer's in 2026, and more than 12 million family members provide their unpaid care (Alzheimer's Association, 2026 Facts and Figures). In a 2024 survey of 250 caregivers, 95% dealt with the same questions asked again and again. Every answer has to be the same calm one it was an hour ago, and someone has to be there to give it.
Ray-Ban Meta glasses already have a camera, open-ear speakers, and microphones. I wanted the glasses to be the memory: look at your daughter, and a quiet voice in your ear says who she is. No screen, nothing to learn.
Impact
This has been tried once before. In 2019 a University of Minnesota team paired a phone worn on a lanyard with a smartwatch that buzzed and showed the name of an enrolled face. In a six-month randomized pilot with 48 people, nothing moved (McCarron, Zmora & Gaugler, JMIR Aging 2019). Their participants said why: it was too slow, so the person had already asked; a name on a wrist was hard to connect to the face in front of them; and the device drew attention. One caregiver asked to hear names through an earpiece "rather than look at my watch." Remi is that earpiece: the name arrives within two seconds, in the ear, from glasses that look like glasses.
The stakes are not only comfort. The 2024 Lancet Commission lists social isolation among the 14 modifiable risk factors for dementia, and in UK Biobank socially isolated adults carried a 26% higher risk independent of loneliness and depression (Shen et al., Neurology 2022). Not knowing who is at the door is how the visits stop. Remi is built for both people in that doorway: the wearer, who never touches a screen, and the family member, who gets a timeline instead of a camera feed.
What it does
Remi lives in a pair of Ray-Ban Meta glasses and a phone in a pocket. The wearer never touches a screen; the family gets the dashboard.
Who is this? When someone the family has added walks up, Remi whispers their name, relationship, and one note within two seconds, panned to the side they are standing on: "This is Sarah, your daughter. She's here to help with the garden." A face not on the roster gets "Someone new is here." A name is never guessed, and each person is announced once per five minutes. Press the capture button on the glasses and Remi says who is in view right now.
Remi only knows the people the family adds. When Sarah walks up, the phone shows the exact words Remi just said, and the family can edit those words on her page.
What were we talking about? Remi hears the conversation through the glasses and keeps the last few minutes. Ask with a tap or "Hey Remi," and it answers from what was actually said, waiting a full second and a half for the wearer to finish a thought.
Every question Ruth asks and every answer Remi gives lands in the day's timeline. "Hey Remi" is an opt-in switch in Settings.
The family's side. One account that syncs between the web and the phone: a timeline of who came by and what Remi said, the people Remi knows, reminders and medication, and Insights, which shows repeated questions and when the hard hours are.
The same day on the web and the phone: what Remi last said, who came by, and the reminders Remi will say next.
The family can hear exactly what Remi says about each person, and Insights shows who visited and which questions came back.
Remi never judges; a repeated question gets the same answer in the same words, and the repetition is logged for the caregiver instead.
Ruth asked "What day is it today?" three times and got the same calm answer each time. The repeat is flagged for the caregiver, never for Ruth.
Memories and Remember. After the introduction, Remi adds one line of memory: "Sarah brought you lemon bars last week and you baked them together." The wearer can record one by voice, or the family can write one, and Claude distills it into a single sentence of fifteen words or fewer that the glasses say. Each memory has a switch for whether Remi says it, and the family can edit the words. Remember is the quiet side of the same loop: Ruth sees one familiar face at a time and is asked, gently, whether she knows them. Nothing is scored. If a name slips away, Remi simply says it, and the session is logged for the family like any other moment.
Remember shows Ruth one familiar face at a time. Nothing is scored, and if a name slips away, Remi simply says it.
Adding a person takes a name, a relationship, a note, and one to three photos. The phone has already indexed the faces in the newest 2,000 photos of the family's library with the same model it uses on the glasses, so as soon as a seed photo is chosen it suggests up to twelve more photos of that person. iCloud originals are never downloaded, and the library index is kept apart from the roster Remi recognizes against.
How I built it
Remi is deterministic first and generative second. Who is in front of the wearer is decided on the phone by cosine similarity against the family's roster. A language model is only called when the user initiates it.
A Swift module inside an Expo app owns the glasses through Meta's Device Access Toolkit 1.0.1: the camera stream, in-band microphone audio, and button. Apple Vision finds faces; an ArcFace ResNet50 pytorch model I ported to Apple's Core ML embeds each one on the Neural Engine in 1.6 ms, and the frame is discarded. The conversion and thresholds are documented in the module README, and every route the phone calls is in CONTRACT.md.
The voice
Every line Remi speaks is ElevenLabs. Latency was the first problem: generation takes 300 to 450 ms plus a download, on top of recognition. So each roster line is generated once and the phone downloads every MP3 during roster sync. A recognition plays from disk the instant a face matches, and the log call to the server happens afterwards, in the background. The same words always sound exactly the same, which matters when the wearer asks the same question four times. A caregiver can put an audio tag like [whispers] in a note; the line is routed to ElevenLabs v4 Turbo, which performs the tag, and the tag is stripped from everything shown on screen. The speech id is the text itself, so any serverless instance can regenerate a line and the phone's disk is the only cache.
Facial Recognition Model
This was one of the most challenging parts of building Remi. I knew when I started that there were some non-negotiables here:
- Live camera stream never leaves the device
- >95% accuracy on faces it knows, 0% false identification
- <0.5 seconds to identification
- No battery drain
These requirements led me to test 7 different open source models against a dataset of over 6,000 labeled face pairs. These tests ran against the very same crop and preprocessing pipeline I use in the mobile app to make sure I chose the right model for my data.
Seven embedders on 6,000 LFW pairs through the phone's exact crop pipeline, at clean, 56 px, and 40 px face sizes. Full tables in RESULTS.md; the method and takeaways are in the evaluation README.
The only model that satisfied my requirements was ResNet50, which is the model shipped in the live demo. ResNet100 does perform slightly better, but I found when I implemented it the battery consumption over long runs was too high, and the phone began to heat up and throttle. ResNet50 achieved 99% accuracy when run against my dataset of 6,000 face pairs, and had 0 misidentifications.
Mobile App
For the mobile app, I decided to use React Native because I'm very familiar with React for UI design. As I was building, I was constantly running into performance issues, so I rewrote the majority of the logic in Swift bundled as a expo module. Swift is much faster for video and photo processing, and sending frames across the JS-Swift interop border is a expensive operation.
Setup asks one question first: who will wear the glasses. A minute later the family is on Today.
Challenges I ran into
An SDK whose stable release is three weeks old. Device Access Toolkit 1.0 shipped on September 30, and Meta's docs say integrations created against the year-long preview do not work with it, so there was little to copy from. DAT 1.0.1 ships dynamic frameworks in a Swift package that React Native will not embed, so I vendor them at pod-install time. There were multiple interfaces that returned conflicting data, so parsing through them to find the accurate data source was a headache.
The first face model felt weak, and the reason was a number. My first iteration shipped a 6.6 MB MobileFaceNet at the 0.50 cosine threshold from InsightFace's README. It worked on a clean photo at arm's length and quietly missed people a few steps away, with no error to chase. Instead of guessing, I built an evaluation that pushes 6,000 labeled LFW pairs through the phone's exact crop and preprocessing, and added two degraded tiers that shrink and JPEG-compress faces to the 56 px and 40 px the glasses stream actually delivers. The threshold was the bug: through my crop, which has no landmark alignment, same-person scores run lower than the published numbers assume, so 0.50 rejected 23% of real matches on clean photos and 70% at 40 px, while strangers never scored above 0.30. With the harness in place I converted and measured seven models, moved the threshold to 0.30, and shipped ResNet50 ArcFace: 99% of same-person pairs accepted, zero strangers in 9,000 comparisons, 1.6 ms per face. The whole weekend's list of surprises like this one is in NOTES.md, 340 lines, each dated.
Open-ear speakers sit centimeters from the mic. With the mic open during playback, the Realtime model heard itself and answered its own echo. To address this, ask Remi is half-duplex: the mic closes from the first audio packet until after the last one plays. Before implementing this fix, Remi stuttered and went into a loop of answering itself.
Accomplishments that I'm proud of
The whole loop runs on real Ray-Ban Meta glasses: a person walks up, and the wearer hears their name in under two seconds. Saying "Hey Remi" opens a live conversation with the last ten minutes of what was said as context. Faces are matched only against the family's roster, on the phone; no frame is uploaded for recognition. Only moments are kept (who, when, what was said), any moment can be deleted for good, and the Privacy page shows live counts of exactly what Remi holds. The caregiver side is a real second user, not an admin panel.
Left: the Live view shows what Remi just said, large enough to read across a room. Right: the Privacy page counts exactly what Remi keeps.
Built solo, between 2:24 PM Saturday and the submission:
| Commits | 56 |
| Lines of code | 21,500 (Swift 5,000; TypeScript 15,000; Python 940) |
| API routes | 28 |
| Web pages | 20 |
| Swift files in the glasses module | 13 |
| Face models converted and evaluated | 7, on 6,000 labeled pairs |
| Database models | 11 |
What I learned
Swift was the first language I ever learned, five years ago from YouTube tutorials, and I had not written it seriously since. This weekend it carried the whole glasses side of Remi: the camera stream, Core ML on the Neural Engine, audio sessions, and the wake word. That was satisfying, but it didn't change how I think.
My biggest takeaway was a clearer picture of where AI actually is. The models in Remi took about a fifth of my time. The other four fifths went into everything around them: when to speak and when to stay quiet, how long to wait for someone to finish a thought, what to do when Bluetooth drops mid-sentence, how to say the same thing the same way twice. None of that is intelligence, and all of it decides whether the intelligence is usable. Most people are nowhere near the ceiling of the models they already have. The gap is no longer model capability; it is the harness. This became especially clear to me developing a product through the lens of someone with dementia. I think in the next 5 years, the biggest innovations are going to be using the AI we already have in unique and novel ways, not just in beating the last benchmark.
What's next for Remi
Near term: a permissively licensed face model (the InsightFace weights are research-only; EdgeFace-S is already evaluated and converted) and a consent flow the wearer understands and agrees to. Mid term: captions on Ray-Ban Display for wearers who are hard of hearing, and a clinician view of repeated questions over weeks, not days. Long term: agitation cues from tone and pacing, Android, and a fully offline mode. I want Remi on a real family before the semester ends.
Open source used
InsightFace ArcFace weights (research license), coremltools, onnx2torch, PyTorch, sherpa-onnx, onnxruntime, Expo, React Native, Next.js, Prisma, Meta Wearables Device Access Toolkit.
Log in or sign up for Devpost to join the conversation.