familiar

What the glasses can see
Blur strangers
Map out a room of people
Create contacts for your friends
Pipeline
Architecture Diagram

Inspiration

What if your glasses could help you recognize a friend before they even say hello?

More than 32 million Americans experience vision loss that glasses or contacts alone cannot correct, according to the CDC.

For people who are blind or have significant vision impairment, everyday social moments can present unexpected challenges. Recognizing a friend approaching, noticing someone joining a conversation, or navigating an unfamiliar environment isn't always easy.

That's why we built Familiar.

"Marcus is approaching."

Three simple words. One opportunity to say hello first.

Inspired by Meta's "Bringing People Closer Together with AI" challenge, Familiar uses wearable AI to make social connection more accessible.

Our goal isn't to replace human interaction. It's to make more of it possible.

What it does

Meet Familiar, your AI-powered social companion.

Familiar connects Ray-Ban Meta smart glasses with Poyo, our conversational voice assistant, to help blind and low-vision users navigate everyday social experiences.

Rather than constantly narrating the world, Familiar delivers the right information at the right moment.

Key Features

Feature	What it does
🤝 Friend Recognition	Recognizes consenting, enrolled friends and quietly announces when they approach.
💬 Remember People	Saves new contacts through a consent-based introduction flow, including useful conversational context.
👁️ Scene Understanding	Answers questions about surroundings, objects, and what's happening nearby.
🚶 Obstacle Awareness	Provides contextual alerts about potential obstacles in the environment.
📱 Social Context	Summarizes available Instagram profile information to help users discover shared interests.
📍 Accessible Discovery	Recommends potentially suitable places using accessibility information, web search, and Yelp reviews.

The Familiar Experience

See → Understand → Connect

Perceive: Ray-Ban Meta glasses capture relevant visual information.
Understand: Familiar recognizes enrolled friends or interprets the environment.
Connect: Poyo delivers a short, natural voice response through the glasses.

Sometimes, knowing who's standing in front of you is enough.

How we built it

Familiar combines on-device computer vision, multimodal AI, wearable hardware, and real-time voice interaction in a native SwiftUI application.

Our Tech Stack

Component	Technology
Wearable Integration	Ray-Ban Meta Glasses + Meta Wearables Device Access Toolkit
iOS Application	SwiftUI
Face Recognition	Apple Vision + ONNX Runtime
Multimodal AI	Muse Spark
Speech-to-Text	Muse Voice Transcribe
Text-to-Speech	ElevenLabs
Social Intelligence	Instagram Data API + Muse Spark
Place Discovery	SerpAPI + Yelp Reviews

Under the Hood

Our architecture follows a privacy-conscious processing pipeline:

Glasses → iOS App → On-Device Recognition → Face Redaction → Muse Spark → Voice Response

Capture: The glasses provide visual input through Meta's Wearables Device Access Toolkit.
Recognize: Apple Vision and ONNX Runtime identify enrolled, consenting friends locally.
Protect: Detected faces are masked before images are sent to cloud-based scene analysis.
Understand: Muse Spark processes face-redacted images and conversation context to generate responses.
Respond: ElevenLabs delivers natural spoken responses through the glasses, while Muse Voice Transcribe handles spoken requests.

For additional context, Familiar integrates Instagram profile summaries and SerpAPI-powered discovery, using accessibility information and Yelp reviews to help identify potentially suitable venues.

Privacy by design: Facial recognition stays on-device, and detected faces are redacted before cloud scene analysis.

Challenges we ran into

Building reliable wearable AI meant tackling challenges beyond simply connecting a camera to an LLM.

Bluetooth and camera conflicts: Activating the glasses' microphone could interrupt the camera stream. We separated audio input and playback to keep both working more reliably.
Response latency: Delayed answers made conversations feel unnatural. We introduced streaming speech to improve responsiveness.
Information overload: Repeated announcements could become distracting. We added rules to prioritize relevant cues and manage interruptions.
Conversation continuity: Independent responses lacked context. Shared conversation memory helped maintain more natural interactions.

Our biggest challenge wasn't just teaching Familiar what to say. It was teaching it when to stay quiet.

Accomplishments that we're proud of

We brought together wearable perception, conversational AI, and consent-based facial recognition into one integrated prototype.

Some highlights:

On-device face recognition for previously enrolled, consenting friends.
Privacy-conscious scene understanding with face redaction before cloud analysis.
Natural voice interaction through Ray-Ban Meta glasses.
Context-aware announcements designed to minimize unnecessary interruptions.
Social and accessible place discovery to support experiences beyond basic object recognition.

But our proudest accomplishment isn't a specific API or model.

It's creating an experience where someone can recognize a friend approaching and have the opportunity to greet them first.

That small moment represents exactly the kind of independence Familiar aims to support.

What we learned

Accessibility is about timing, not just information.

More information doesn't always mean a better experience.

A wearable assistant could describe everything it sees, but constant narration can quickly become overwhelming. Sometimes a short, timely cue is far more useful.

We learned that privacy, hardware reliability, user control, and thoughtful interaction design matter just as much as model capability.

Most importantly, assistive AI should empower people to make their own decisions rather than make those decisions for them.

The best assistive technology doesn't demand attention. It creates opportunities.

What's next for Familiar

Our vision is to evolve Familiar from a hackathon prototype into a more complete AI-powered assistive wearable for blind and low-vision users.

Our Roadmap

Priority	What's next
User Testing	Validate the experience with blind and low-vision users.
Better Perception	Improve recognition reliability and obstacle awareness.
Personalization	Expand consent-based contact memory and user preferences.
Accessible Discovery	Strengthen accessibility-aware recommendations and verification.
Creative Expression	Explore voice-controlled Muse Image generation to help users express ideas visually.