Outlens

Point at any object: Outlens outlines it, estimates its color, names it and says it out loud, then recolors it so people with color blindness can tell colors apart.

Comment

Inspiration

About 1 in 12 men and 1 in 200 women have a color-vision deficiency. Everyday questions ("is this shirt navy or black?", "which cable is which?") still mean asking someone else. Color filter glasses exist, but they can't tell you what you're looking at. I wanted to preview the software for future smart glasses: look at something, ask, and get an answer.

What it does

Outlens is a phone and laptop preview of a "glasses view":

  • Tap (or pinch) an object. Outlens freezes the frame, draws a glowing outline around the object and estimates its color from the pixels inside it.
  • Gemini names it.Only a small crop around the selected object is sent, and it comes back as something like "mug: mostly dark blue with a white stripe near the top." Tap More for a longer description: which parts are which color, and any print or pattern.
  • ElevenLabs says it. Tap 🔊 Listen to hear "Mug. Estimated color: dark blue." in a natural voice. The browser voice is a fallback.
  • Two demo users, two kinds of color blindness. Maya (deutan, red-green) and Leo (tritan, blue-yellow). The whole camera view can be shown as Maya sees it (a simulation), so people with typical vision understand the problem.
  • ✨ Boost. Recolors just the selected object (daltonization) so its colors become easier to tell apart for that user's type.
  • 🎤 Ask. Voice commands: "what color is this?" (reads the center of the view), "boost", "more", "switch to actual vision" and more. Ask about a traffic light, and it explains how to read one by position instead of guessing the color.
  • ✋ Pinch (prototype). Pinch your thumb and index finger in front of the camera to select, like you would with glasses. Tapping always works too.

Every color is labeled "estimated" and every simulation is labeled as one. Outlens makes no claims about food ripeness, safety or health.

How I built it

  • Next.js 16 (App Router), React 19 and Tailwind v4, deployed on Vercel, with a custom domain registered with GoDaddy.
  • Outline: a custom "magic wand" in CIELAB color space. It grows a region from the tap point over similar colors, corrects for shadows, and tries strict to loose thresholds while detecting when the region "leaks" onto the table. It runs in about 50 ms on-device, with no model. The mask is traced into a smooth outline (loop tracing plus Ramer–Douglas–Peucker simplification).
  • Callout: the label is placed automatically in free space, with an elbow leader line that never covers the object.
  • Color-blindness simulation: Machado et al. (2009) matrices in linear RGB, run in a WebGL shader on the live video. iPhones ignore CSS filters on video, so the shader was required.
  • Boost: a daltonization matrix (I + shift·(I − simulation)) applied only inside the object's outline.
  • Gemini API: gemini-3.5-flash-lite via generateContent, with a JSON schema reply ({name, description}), a server-side rate limit and a filter that drops any ripeness or health wording.
  • ElevenLabs API: text-to-speech through a server route that only accepts phrases shaped like "Object. Estimated color: x.", to protect credits.
  • MediaPipe Hand Landmarker: on-device hand tracking for the pinch gesture.
  • Web Speech API: voice commands.
  • Design: the landing page was designed with Figma Make. The app uses a navy, amber and blue palette, with no red/green pairs anywhere in the UI. Buttons are at least 48 px and text at least 18 px.
  • API keys live only in server environment variables.

Challenges I ran into

  • Outlining objects without a heavy model. The MediaPipe interactive segmenter took about 12 s per tap on CPU and returned empty masks on GPU, so I replaced it with my own region-growing algorithm and tuned it on real objects (btw white plates on white tables were the worst choice for demonstartion).
  • The simulation didn't show on iPhone, because Safari ignores CSS/SVG filters on <video>. I rewrote it as a WebGL shader.
  • Gemini latency. The first model I tried timed out. I switched to a lighter model with minimal thinking, and naming now takes about a second or two.
  • Voice on iOS. Audio has to be unlocked inside a tap, and the reply has to wait until the microphone closes.
  • I did all of this solo, as a beginner, in 28 hours.

Accomplishments that I'm proud of

  • It works on a real phone: tap a mug, and within seconds you see it outlined, named and recolored, and hear it spoken aloud.
  • The simulation toggle makes people with typical vision say "oh, that's what it's like."
  • The privacy design (below).

What I learned

How color-vision deficiency actually works (cone types, why red-green is most common), color science (CIELAB, linear RGB, daltonization), WebGL shaders, and how to design for accessibility first.

What's next

  • An onboarding flow (already designed, not yet built): create a profile, choose or upload your color-vision type, and pick voice preferences. It asks only for a name, email and vision type: no age, gender or location.
  • Patterned and multi-color objects.
  • Running on real smart glasses with an always-on pinch.

Privacy

  • Camera frames are never stored. Only when you select an object is a small crop (max 384 px) around it sent to Gemini. Note: on the Gemini free tier, Google may use it to improve its products.
  • Pinch detection runs on-device (MediaPipe sends Google usage metrics, not images).
  • The color-vision profile stays in your browser.

Built With

Share this project:

Updates

Submission history