Inspiration
More than 338 million people live with moderate-to-severe vision impairment or blindness: 43 million are blind and 295 million have moderate to severe visual impairment, by the IAPB Vision Atlas figures NaviLens summarizes in its global map of visual impairment. These are not people who need reading glasses. They are people for whom a label, a curb, a face across the table and a photo in the family chat are all closed off at once.
There is no single tool that gives them the depth of assistance sight gives everyone else all day. A camera app describes a picture when asked. A reader reads a page when asked. A volunteer service connects a call when asked. Each stops there: nothing watches the street for you, nothing remembers what it just read so you can ask about it, and nothing tells you what the people you love shared today in a way you can picture. The cost is not only independence. It is being the one person at the table who did not see the picture everyone is laughing about. iSight was built to close that gap: everyday awareness, independence and connection, in one assistant, on glasses and a phone.
What it does
You wear Meta Ray-Ban Display glasses and talk to iSight. It answers through the glasses' speakers.
See and understand. "Read this" reads the text in front of you from the live video and takes a sharp photo when the print is small; then "when does it expire?" or "what is this book about?" is answered from the text just read. "What's in front of me" describes the scene. "What is this" answers about the thing you hold. Colours, barcodes, weather and the time.
Stay safe. While you walk, the phone runs an object detector and a per-vehicle tracker on the glasses' video and says "Vehicle moving toward you from the right" only when a car is actually turning toward you. A depth model adds "Something large ahead, possibly in your way." All of it on the phone. iSight never says "safe" or "clear", and a spoken "stop" cuts any sentence instantly.
Stay connected. "Who is this?" names a person who agreed to be remembered; "how do they look?" reads their expression. Circles are private family groups: relatives post photos and captions on a web page, an AI describes each photo, and you say "catch me up" to hear the day as one paragraph. "What did Mom post?" answers for one person; "post this" shares a photo from the glasses back to the family. A helper can join a live call and see what you see. Controls also work silently from the glasses' display.
What iSight can do, and why each one is there
- Read text, then ask about it. "Read this", then "when does it expire?" Apple Vision on the phone, a standalone photo for small print; follow-ups are answered by the planner from the text just read. Why: labels, receipts, mail and menus without handing the page to someone.
- Describe the scene. "What's in front of me." One frame to the model gateway. Why: everyday awareness of a room, a street, a shelf.
- Identify the thing in your hand. "What is this." The phone's detector finds the target, one frame goes to the gateway. Why: independence in the kitchen, the pharmacy, the shop.
- Colours. "What colour is this." On the phone. Why: choosing clothes, matching things.
- Barcodes and products. "Scan this." Vision's barcode reader on the phone, product look-up on the backend after consent. Why: shopping without a sighted helper.
- Weather and time. "What's the weather", "what time is it." Time and date on the phone, weather through Open-Meteo. Why: the small facts of a day.
- Vehicle warnings. Nothing to say: it watches. YOLO26n, a per-vehicle tracker with head motion, and a closed set of safety rules, all on the phone. Why: safety at curbs and crossings with no network needed.
- Obstacle warnings. Depth Anything V2 Small on the phone. Why: a large object in your path, spoken once.
- Stop, cancel, end. "Stop." Deterministic, on the phone, no model. Why: trust; the wearer always wins, instantly.
- Attention levels. "Be quiet" through "narrate." On the phone. Why: the wearer chooses how much iSight says.
- Who is this, how do they look. ONNX Runtime on the phone (YuNet, SFace, an expression model); enrolled people are stored on the phone. Why: connection; knowing who walked up and how they seem.
- Circles: the family's day. "Catch me up", "what did Mom post." Circles store, story synthesis, grounding guard; photo descriptions through the gateway. Why: hearing the family's pictures and words as one story.
- Circles: post back. "Post this." A photo from the glasses to the family. Why: the wearer is a poster, not only a listener.
- Live help. "Call Sarah for help." LiveKit; the helper sees the glasses' view in a browser. Why: a person when a model is not enough.
- Silent controls. Buttons on the glasses' display. Why: discretion; no talking out loud in a quiet room.
How I built it
Three parts and one rule.
The glasses (Meta Wearables Device Access Toolkit 1.0.0) stream video at 15 fps, take standalone photos, carry the microphone and speakers, and show a small palette on the display for silent controls.
The phone (Swift 6, iOS 26) is where iSight lives.
- Words. Apple's on-device SpeechAnalyzer transcribes. A deterministic front door (a grammar, no model) claims the controls and every request it recognizes: reads, people, Circles, weather, time, colours. What it cannot place goes to a planner client that works over typed tools in a bounded loop and remembers what iSight just read, so follow-up questions are answered from facts.
- Frames. A perception scheduler runs the safety work first: YOLO26n through Core ML on every frame, a multi-vehicle tracker with head-motion evidence, then a closed set of safety rules. Beside it: Depth Anything V2 Small for obstacles, Apple Vision for text, and ONNX Runtime for face detection, recognition and expression, with enrolled people stored on the phone.
- One voice. Every sentence, from any feature, goes through one output arbiter that owns priority, interruption and cancellation. A warning can cut a story. "Stop" cuts everything. No feature speaks on its own.
The backend (Python, FastAPI) does only what the phone cannot do well.
- A model gateway over OpenRouter with qualified pools: each pool is an ordered chain of models that passed a qualification run of their own (GPT-5.4-mini → Gemini 3.7 Flash → Qwen3-VL for fast visual work; separate chains for precise visual questions and for conversation). Fallbacks are ordered, every call carries a cost reservation against a daily budget, and a usage ledger records the spend in micro-dollars. Every image sent is task-scoped: one frame or one photo for the question asked.
- A decision model (TypeSafe's Jev) types an unclaimed utterance in about a fifth of a second and races the language-model planner; when it is confident about a capability intent, iSight acts without waiting. Composed or long requests skip it and go to the planner whole.
- The planner endpoint turns a request plus facts into typed tool calls.
- The Circles service: SQLite, sealed media, invitation links, a member web page, a story writer, and a grounding guard that checks every generated story against the posts it came from. Any name, number or contact detail that is not in the posts, or a member left out, rejects the story and a verbatim template speaks instead. Per-person answers are templates, never generations.
- A live help call over LiveKit, and weather from Open-Meteo.
The rule: local first, honest always. If it can run on the phone, it does; if the network is gone, warnings and reads keep working; if iSight is not sure, it says so or says nothing.
Challenges I ran into
- Bluetooth is slow. A full-resolution photo took over 20 s to cross the glasses' link; 720p takes 3 to 6 s. iSight reads the live frame first and asks for a photo only when the frame is not legible, and re-encodes the photo on the phone because the server refused the glasses' container.
- The stream does not come back. After a photo the video stream cannot be restarted on this firmware, so the camera is torn down and re-attached every time, under a watchdog.
- Knowing when not to speak. A car passing is not a car coming at you. The tracker needs a per-vehicle history and the wearer's head motion before it is allowed one sentence, and silence must never be mistaken for "all clear".
- Trusting generated text. The first story the model wrote invented a detail. The grounding guard exists because of that: it failed its first live evaluation, I fixed the check, and it passed 3 of 3 after.
- Filming a voice assistant. You cannot record the glasses' speakers cleanly. I built a filming mode that saves every line iSight says as a clean audio file on the phone as it is spoken.
Accomplishments that I'm proud of
- Every capability in the video was exercised on the glasses the night before submission: reads with follow-ups, vehicle warnings on a real street, face recognition and an expression cue, the synthesized family story, and a sharp photo posted by voice.
- One arbiter owns every sentence, so a warning can cut a story and "stop" cuts everything, and it has never spoken over itself.
- The Circles story is generated text that a blind person can trust: a deterministic guard checks every name, number, contact detail and member against the posts before it is spoken, and a verbatim template takes over when the check fails.
- About 1,700 automated tests across the phone package, the app and the backend, plus contract schemas for every message between them.
- A filming mode that captures iSight's own voice line by line, which is how the video's narration was made.
What I learned
- Latency is the product. A warning two seconds late is noise. Running detection, tracking and the safety rules on the phone, and keeping speech recognition and the controls local, mattered more than any model choice.
- Trust is binary. An assistant that invents one detail is never believed again, so I spent my time on freshness, cancellation and grounding rather than on prompts.
- Routing is a design problem, not plumbing. A deterministic front door that claims what it knows, a small decision model racing the planner, qualified model pools with ordered fallbacks and cost reservations: that is what made the assistant feel instant and predictable.
- "Perceive a lot, speak very little" is a constraint. The hardest feature to build was the one that stays quiet.
- Hardware has opinions. Photo transfer over the glasses' link, a video stream that will not restart, a camera that can wedge: half of the engineering was living with the device honestly instead of pretending it behaved.
What's next for iSight
See and understand
- A local scene-description model on the phone, so "what's in front of me" keeps working with no network.
- Continuous reading: follow a page or a shelf as the wearer moves, and answer questions across several reads.
- Barcodes with wearer coaching ("a little closer") and product answers that lead with what matters: name, size, allergens.
- Handwriting, currency and screens as first-class reading targets.
Stay safe
- Pedestrian-signal reading at crossings (designed; paused for the hackathon).
- The glasses' motion sensor for head-motion evidence, replacing the image-only estimate.
- Distance and time-to-arrival words once they are measured on the device, never before.
- Finish direction-only guidance to a named object ("guide me to the chair") with the safety rules in the loop.
- Field testing on real streets with blind and low-vision users, and publishing the on-device latency numbers.
Recognize people
- A consent-first enrolment flow so recognition can leave development-only status: the person enrols themselves, and can be forgotten in one sentence.
- Richer expression cues and "who is in the room" summaries, spoken only at the wearer's chosen attention level.
- Recognition that survives lighting, angle and distance, measured before it is promised.
Stay connected
- An encrypted Circles server with real invitations and member accounts.
- Import from the family's existing chats, with consent, so nobody has to learn a new app.
- Voice replies to a specific post, reactions read back, and reminders when someone the wearer cares about has posted.
- Helper alerts over the channels families already use, and a helper page that works from any phone.
Platform
- The same core on other glasses and phones; the platform layer is already isolated.
- Attention levels and quiet hours tuned with users, not guessed.
Track notes
Bringing people closer together with AI
Vision loss isolates in quiet ways. The family shares a photo and everyone reacts but you. A friend walks up and you cannot tell who it is until they speak, or how they feel until they say. In a quiet room you do not want to ask a device out loud what is on the table. Every one of those moments is a small step out of the room, and they add up to feeling left out of your own life. iSight was built around closing that distance. Circles turns a family's scattered pictures and captions into the day, told to someone who cannot see them, and lets them answer back with a photo of their own, so the blind member is a poster and not only a listener. "Who is this" and "how do they look" give back the part of a conversation that faces carry. Silent controls on the glasses' display let the wearer stay discreet in company. A helper can join a live call and see what the wearer sees when a model is not enough. The AI is essential to all of it: it describes every photo, writes the day's story, answers questions about what was read and seen, and does so behind a grounding guard, so every word the family hears about itself is checked against what the family actually posted. These are not the whole answer, but they are large steps from the current gap in information toward a seamless place in everyday life, without the feeling of missing out on so much.
Best use of the Gemini API
Gemini 3.7 Flash is a qualified model in every one of the model pools (scene descriptions, visual questions, conversation and the Circles story), behind a gateway that handles ordered fallbacks, cost reservations and a daily budget. Every image it receives is task-scoped: the one frame or photo for the question that was asked.
Demo notes
Every iSight line heard in the video is iSight's actual output. The glasses' speakers cannot be recorded cleanly, so I used the app's filming mode, which saves the raw synthesized audio of each spoken line on the phone as it is spoken, and placed those source files on the video's timeline so the answers can be heard properly. Every capability shown was exercised on the glasses.
Log in or sign up for Devpost to join the conversation.