Inspiration

Reading isn't really about seeing text. When a sighted person picks up a medicine box, they don't read it. They glance for the dose and the warning, and ignore the other two hundred words.

Most accessibility tools do the opposite. They read everything, or describe the scene ("a white box on a table"), and leave the sorting to the person who can't see it. We wanted to build the glance: the one thing you'd want to know, said first and out loud.

What it does

SightLine is an AI copilot for blind and low-vision people, and for anyone facing text they can't read. Point your phone, or a small camera on your glasses, at a menu, a can, a medicine box or a sign. It says what matters in about 30 words: the name, the dose, the allergens, the price.

Then you can ask for more: "How much is the salmon?" or "Was that medicine recalled?" It also scans barcodes, checks the web for recalls, translates 13 languages, and remembers what you scanned so you can ask about it later.

How we built it

  • Claude (Haiku 5.5) reads each photo and returns a structured answer: a short spoken line first, details in sections.
  • ElevenLabs gives it a natural voice and hears your questions.
  • Bright Data searches the web live for recalls, nutrition and reviews.
  • Neon Postgres caches answers and holds your scan history, which a Claude agent searches when you ask about it.
  • Next.js on Vercel runs the website, the web app and the API. The iPhone app is React Native with Expo, and the glasses camera is an ESP32-S3 board. All three use the same backend.

Challenges we ran into

  • Focus. Our hardest problem, and one we're still tuning. A camera sees the shelf, the poster and the hand, and has no idea which one you meant. Now every answer starts by deciding what you're pointing at and ignoring the rest, and the glasses only read something that holds still for two frames in a row.
  • Speed. A spoken answer that takes 8 seconds feels broken. We picked the fastest model that still read accurately, cached repeats, and never let the web search hold up the first answer. A photo takes about 3 seconds.
  • Sound. Text that reads well can sound wrong. "6.50 dollars" came out as "six. fifty dollars" until we rewrote prices and units for the ear.
  • Honesty. On a medicine label, a confident guess is worse than no answer. SightLine says when it can't read something and how to fix the shot.
  • Venue Wi-Fi. The campus network blocked phone-to-laptop traffic, so we moved everything to the live site.

What we learned

  • The hard part of reading for someone isn't recognizing the words. It's knowing which ones they wanted.
  • In a spoken interface, what you leave out matters more than what you put in.
  • An accessibility tool earns trust by saying "I'm not sure".

Built With

Share this project:

Updates

Submission history