Inspiration

A trip to the grocery store starts long before you reach the shelf. You need to work out the route, find the bus stop, check which bus is arriving, and then find the product you came for. For someone who is blind or has low vision, information along that journey can be difficult to access visually, from the number on a bus to the flavor printed on a bag of chips. We wanted to bring those moments together. Someone could ask, “How do I get to the store?”, hear a route and departure estimate from Google Maps, and later ask, “What bus number can you see?” or “Is this the Doritos I’m looking for?” That became BrightBlind: a wearable companion that connects spoken questions with visual information, transportation data, and remembered context.

Our one minute demo focuses on a bag of Doritos. It is one small part of the bigger idea: making useful information easier to access throughout an ordinary day.

What it does

BrightBlind connects a Vexo wristband and Meta glasses through an AI system built around voice, using Pipecat and Deepgram. The wearer asks a question. The band captures it, the backend decides what information is needed, and the glasses provide the camera view and play the answer. For our demo, the task is finding and identifying the right bag of Doritos. Beyond that example, BrightBlind can help with:

Groceries: identifying products and reading visible labels.

Finding things: looking for food or another requested item in a room.

Transportation: getting a Google Maps transit route, departure stop, and bus departure estimate.

Understanding a scene: describing visible potential obstacles, such as chairs, bags, boxes, or stairs.

Remembering context: retrieving information from earlier conversations and observations.

Say “BrightBlind” to get its attention, then ask your question. It replies through the glasses, so you can keep your hands free.

Vexo keeps listening and remembering, so the Meta glasses only need to capture photos or stream when a task calls for it. That makes the band central to how we manage battery use, reducing unnecessary camera activity between requests. BrightBlind is designed to complement established mobility aids. It can describe what appears in a camera view, but that does not mean it can guarantee a clear path or safely tell someone where to step.

How we built it

Pipecat coordinates the voice pipeline, and Deepgram handles speech recognition and speech generation. They turn the wearer’s spoken request into something the backend can act on, then turn the result back into speech. Google Gemini 4 interprets requests and analyzes camera images. Google Maps Routes API supplies transit information, including bus lines, departure stops, and estimated departure times.

We divided the backend into a few focused components:

  • A planner decides whether a request needs vision, transit information, or memory.
  • A vision agent examines selected camera images and returns observations.
  • A transit agent gets route and departure information from Google Maps.
  • A memory layer keeps useful context from speech and visual observations. The devices and backend communicate over WebSockets. Slower work, such as analyzing an image, runs separately from the voice connection. That lets the assistant acknowledge a request while it is still checking the view.

We built the backend with Python, FastAPI, and SQLite, and deployed it on Railway. We also made a dashboard that shows transcripts, selected frames, agent decisions, and provider activity. It helps us understand what happened when something works, and when it doesn’t.

Challenges we ran into

The biggest challenge was latency. An answer can be correct and still arrive too late to feel useful. We had delays at several points: taking the photo, transferring it, analyzing it, generating speech, and sending that speech back to the glasses.

We measured those stages separately instead of treating everything as “the AI is slow.” We used low thinking for quick Gemini tasks, routed obvious commands directly, reused active camera streams, and compared Deepgram speech output options to shorten the wait. Getting the wearables to work together was another challenge. The band and glasses use different connection paths and have different limitations. We had to work through camera startup, Bluetooth photo transfer, reconnects, and audio routing. A successful API call on a laptop was only one piece of the experience.

We also had to decide how to handle uncertainty.

Seeing a chair does not tell us whether there is enough room to walk past it. Reading a bus number does not establish that boarding is safe. We kept those distinctions in the product: describe the evidence, acknowledge uncertainty, and discard results that are no longer current.

Accomplishments that we're proud of

We got Pipecat, Deepgram, Gemini, and Google Maps working together through real API calls. We verified speech recognition and speech generation through Pipecat, image analysis with Gemini, and live transit requests through Google Maps. A test with the connected glasses returned a photo, produced a Gemini observation, and delivered generated Deepgram audio back to the device app. We are also proud of making the system inspectable. We can see which component acted, what information it used, and where time was spent. The Doritos demo gives us a simple way to show that work. It turns a lot of technical pieces into a familiar question: “Is this the item I wanted?”

What we learned

For a wearable assistant, timing matters as much as the wording of the answer. We learned that choosing a capable model is only part of the job. Camera delays, connection stability, speech playback, and response length all affect whether the interaction feels useful. Sometimes a shorter answer or avoiding a camera restart made a bigger difference than changing the model. We also learned to keep the demo focused. One everyday task is easier to understand and evaluate than a long list of features. What's next for BrightBlind

Our next priority is testing with blind and low vision users and orientation and mobility specialists. We want to learn which descriptions are helpful, when the assistant should speak, and when it should stay quiet.

On the technical side, we want to improve streaming audio playback, reduce camera delays, strengthen reconnect behavior, and measure latency and battery impact during real wearable sessions.

We want BrightBlind to become a dependable companion for everyday questions, from planning a journey to finding something on a shelf, with clear limits on what it can confidently tell the wearer.

Built With

Share this project:

Updates

Submission history