Inspiration
Familiar started as a tool for people who forget names at networking events. Around noon we changed it to help people with dementia, and their families, instead. A forgotten name at a mixer is awkward. For someone with dementia, not recognising their own daughter is distressing for both of them. Changing the audience also fixed our consent problem: family members register themselves, so the only people the system recognises are people who agreed to it.
What we wanted was simple. When someone walks up, the wearer hears: "This is Maria, your daughter. Last time, she told you about her grandson's graduation."
What it does
Familiar uses two wearables, one iPhone app and one small server.
- The Vexo Open band is the ears. It sits on the wrist and streams Opus audio (16 kHz, 50 packets a second) to the iPhone over Bluetooth.
- Ray-Ban Meta glasses are the eyes. The camera sends a picture every 0.4 seconds, and the server checks it against the faces of registered people. The iPhone camera works as a fallback.
- The face says who; the band says what. If a voice speaks while exactly one registered person is in view, it gets that person's name.
- Gemini remembers. When a conversation ends, Gemini reads the named transcript and writes, for each person, a "Last time, …" sentence, lasting facts, and things to ask next
- The recall moment. When a registered person shows a card with their name and relationship andreads it out loud.
How we built it
- iOS app (Swift, SwiftUI, CoreBluetooth). The kit has no Swift library, so we wrote the band's Bluetooth protocol as a Swift package. We tested it against the kit's test our team's band. Meta's Wearables Device AccessToolkit gives us the glasses' camera. The screens come from a React prototype we designed first.
- Python WebSocket server. It holds every API s, and the app pairs with it using a 6-digit code.It forwards the band's audio to Deepgram for live transcription with speaker separation. It also matches faces with OpenCV (YuNet to find a face, SFace to recognise it), pic with NVIDIA NeMo TitaNet running on sherpa-onnx,and sends finished conversations to Gemini.
- Who said it. Deepgram often hears the wearerof them as one voice, so the server decides "thewearer or the other person" itself, phrase by phrase. Each phrase gets three votes between −1 and 1, and the votes are added:
- Privacy rules in the code. Only registered perver refuses to register anyone without a confirmed yes. Words from people who aren't registered never reach the phone, are never saved, and are never sent to Gemini. Camera pictures are compared and then discarded. Faces anstored as numbers, never as photos or sound.
Challenges we ran into
- One microphone, two people. For a whole testabelled the wearer and their partner as "speaker 0". That's why we built the phrase-by-phrase voting above.
- A voice model that worked only sometimes. OuSpeaker CAM++) gave correct output only for clipsthat were an exact multiple of 2 seconds long, which meant the registered voice itself was unusable. We switched to TitaNet, which works from about half a second.
- A silent failure. After the model switch, the server rightly refused the old registered voice. Every conversation for the next hour then ran as "no voice registered", and wng the server log line by line.
- The camera's mouth measure did worse on real glasses than in tests. On our test photos, a still mouth scored about 1 and a talking one about 2. On the real glasses it stayoever was talking, and giving it a full vote swapped the two speakers. Now the camera can only nudge the decision until it has proved itself in that conversation.
- Names flipping. As the wearer looked around, between two people's names. Now votes are weightedby how many words were said, and a name changes only on a clear lead.
- Hardware logistics. The venue Wi-Fi blocked d laptops, so the server ran on a laptop joined tothe iPhone's hotspot. Getting the glasses working took a firmware update and Developer Mode, and their stream sometimes ended with "critical error", so the app now restarts it
Accomplishments that we're proud of
It works end to end on real hardware. The band's mm and came back as text split by speaker. TheRay-Ban Meta camera recognised registered faces and put their names on the right lines. Gemini wrote a profile when the conversation ended. In our last real test, with thed, every line of the conversation went to the right person.
What we learned
- A wrong name is worse than no name. Telling someone with dementia that a stranger is their daughter does real harm, so every match is strict, and when the system isn't s
- Don't trust any single signal. Voice, loudness and the camera each failed on their own at some point during the day. Checking them against each other is what made the
- Loudness isn't direction. One microphone can't hear where a sound comes from. Loudness at the wrist is a decent stand-in for distance, but a hand in a pocket breaks it.
- Each device should do what it's best at. The band can listen all day on a small battery. The glasses' camera drains its battery fast, and microphone access in Meta's tool. So the band listens and the glasses look.
- Read the logs. Every real bug today showed up first as a line of numbers in the server's window.
What's next
- Check that the spoken line plays through the glasses' speakers, and give it a calmer voice.
- A caregiver view, so family can set up people ans visits.
- Waking the glasses' camera only when the band hears someone other than the wearer, to save battery.
- Testing with more people, in noisier rooms, over
Built With
- avfoundation
- bluetooth
- corebluetooth
- deepgram
- ffmpeg
- gemini
- ios
- meta-wearables-device-access-toolkit
- numpy
- nvidia-nemo
- onnx
- opencv
- opus
- python
- ray-ban-meta
- react
- sherpa-onnx
- swift
- swiftui
- typescript
- vexo-open-kit
- vite
- websockets
- xcodegen