Lumi

Lumi transforms a phone camera into spoken guidance for blind and low-vision users, identifying surroundings, explaining spatial relationships, and guiding everyday tasks step by step.

Comment

Inspiration

Most visual-assistance apps for blind and low-vision users describe a photo: "A desk with a mug, some books, and a lamp." That's useful, but it doesn't help someone actually do anything. Finding the mug, picking it up without knocking it over, and knowing when the desk is really tidy all need a back-and-forth conversation grounded in the space the person is standing in.

We started with a gamified idea where everyday chores became voice-guided quests. As we built it, we realized the XP, levels, and rewards got in the way of what mattered: calm, trustworthy guidance. So we removed the game layer and rebuilt the project as Lumi, which turns a phone camera into spoken, step-by-step guidance that works by touch and sound.

What it does

The user points their phone at the room and says "Hi Lumi, help me tidy my desk." After one tap to start, everything happens by voice.

  • Perceive: the phone streams one camera frame per second. While a task is running, Lumi analyzes the newest frame, looking only for what the current step needs.
  • Understand: a Vision Agent finds objects, says where they are relative to the user's body ("at your two o'clock, about an arm's length away"), and flags hazards like hot or sharp items.
  • Guide: a Quest Agent builds a task plan from objects the camera actually saw, then speaks one short step at a time. Directional beeps speed up and pan toward the target.
  • Verify: a step only counts when the Vision Agent answers a yes/no check with PASS. A FAIL or UNCERTAIN result leads to a spoken correction instead of a false "done."
  • Map: a laptop dashboard shows what Lumi sees, the current task, and an explorable 3D model of the room that clearly labels what was observed, what was estimated, and what is unknown.

Users can also ask questions like "Is the room clean?" or "Where are my headphones?" For 20 seconds after Lumi speaks, follow-up questions don't need the wake phrase.

We designed it for two people: the blind user, who uses it every day independently, and the caretaker or family member, who sets it up once by scanning a QR code.

How we built it

There are three pieces:

  1. Phone controller (HTML, JavaScript, Web Audio): camera capture, a voice detector that only sends audio when someone is speaking, spoken narration, and stereo-panned beeps. It's a web page, so there's nothing to install.
  2. Camera server (Python, FastAPI): serves the phone page over HTTPS on the local network, receives frames over WebSockets into a ring buffer, and forwards game traffic. It holds no API keys.
  3. Game API and dashboard (Node, TypeScript, Express, React, Vite, three.js): runs the two agents, task state, step verification, and the laptop dashboard. It's the only process that calls external APIs, and it only listens on 127.0.0.1.

Meta Muse Spark powers both the Quest Agent and the Vision Agent. Muse Voice Transcribe handles speech-to-text, and we pass it the agent's name and the task's object names as keywords. ElevenLabs (eleven_flash_v2_5) provides the narrator voice, with on-device speech as a fallback.

Honest 3D from one camera

A single phone camera has no depth sensor and no position tracking, so we don't pretend to reconstruct the room perfectly. We place each object along the ray through where it appeared in the frame. For an object whose bottom edge sits at an angle ( \theta ) below the horizon, with the camera at height ( h_c ) and a table at height ( h_t ), the distance falls in this range:

$$ d \in \left[ \frac{h_c - h_t}{\tan\theta},\ \frac{h_c}{\tan\theta} \right] $$

The near end assumes the object is on a table, and the far end assumes it's on the floor. We show this as a range, not a single point. The left-right direction comes from the box center: ( \phi = (x_c - 0.5)\cdot \text{FOV}_h ). Anything outside the camera's roughly 65° view is drawn as unobserved, never invented.

Verified progress

The Vision Agent's confidence ( c ) decides whether a verdict stands:

$$ \text{verdict} = \begin{cases} \texttt{PASS} / \texttt{FAIL} & c \ge 0.55 \ \texttt{UNCERTAIN} & c < 0.55 \end{cases} $$

Only a PASS can complete a step, so the system would rather ask the user to pan slowly than claim something it can't see.

Challenges we faced

  • Phone cameras need HTTPS. Mobile browsers block camera access on plain HTTP, even on the local network. We generate a self-signed certificate for the laptop's network IP so the phone can connect by scanning a QR code.
  • AI models say "done" too easily. Early versions marked steps complete when they weren't. Splitting the work into a Quest Agent that plans and a separate Vision Agent that verifies, with a confidence floor, fixed this.
  • Speed and cost. We send one frame per second but analyze only the newest frame every 2.5 seconds. Requests from the user jump ahead of background frame analysis, and the microphone stays muted while Lumi is speaking.
  • Background chatter. Nearby conversations kept triggering commands. Starting a new task always needs the wake phrase, and follow-ups only skip it for a short window.
  • Being honest about 3D. It was tempting to draw a polished full room. Instead, every object is labeled observed, estimated, or seen earlier, and objects the task expects but the camera hasn't seen are listed, not drawn.
  • Keeping keys safe. The phone talks only to the laptop, and API keys never leave the game API. They're sent in headers and removed from logs.
  • Showing the laptop what the phone sees. The dashboard originally showed only frames the AI had analyzed, so with no task running the laptop stayed blank. We added a live preview that hands off to analyzed frames once a task starts.

What we learned

  • For accessibility, trust matters more than features. Saying "I'm not sure, pan slowly" beats a confident wrong answer.
  • Directions should be given from the user's body (clock positions and arm's length), not as screen coordinates.
  • Two specialized agents, one that plans and one that verifies, are much more reliable than one agent doing both.
  • Simplifying the product (removing gamification and collapsing many screens into one dashboard) made both the demo and the experience better.

What's next

  • Using phone depth sensors (LiDAR) and position tracking for real multi-angle 3D mapping.
  • Running fully on the phone, without the laptop.
  • Optional caretaker updates when a task is finished, sent only with the user's confirmation.
  • Testing with blind and low-vision users and orientation-and-mobility instructors.

Lumi supplements a cane, a guide dog, and mobility training. It never replaces them.

Built With

Share this project:

Updates