SightLine

• 

• 

• 

Inspiration

Most visual-assistance apps answer a question about a single image: “What is in front of me?” That is useful, but it stops before the hardest part—helping someone safely complete the actual task.

SightLine started with a different question:

What if an AI could stay with the user through the task, give one clear instruction, verify that it worked, and interrupt when the scene became unsafe?

That became SightLine’s core loop:

Perceive → Reason → Guide → Verify

What it does

SightLine is a voice-first visual copilot for blind and low-vision people.

A user gives it a real-world goal such as:

• “Help me make coffee.”
• “Guide me to the door.”
• “Read this label.”
• “What am I holding?”

SightLine observes the camera, provides one concise spoken action, waits for the user, and verifies the visible result before advancing.

A separate background safety watcher scans the scene every five seconds. It does not silently change the task, but it can interrupt immediately when it detects a potential hazard.

The MVP includes four focused modes:

• Guide me: Complete a physical task step by step.
• Orient me: Understand a room and move toward a nearby destination.
• Read this: Read useful text from labels, signs, menus, and mail.
• What is this?: Identify an object, its orientation, and useful controls.

SightLine also supports voice commands including “done,” “repeat,” “pause,” “resume,” and “end.”

How we built it

SightLine is a mobile-first Next.js PWA built with TypeScript, React, Codex, and the OpenAI Responses API.

The browser captures an on-demand frame from the rear camera and resizes it to reduce latency and cost. The frame is sent to GPT-5.6 together with:

• The user’s goal
• The current task step
• A short rolling memory of recent observations
• Whether the request is a task turn or a safety-only scan

GPT-5.6 returns strict structured output containing the current observation, spoken instruction, confidence, task state, next physical action, and hazard status.

The browser then speaks the instruction and uses visual and haptic feedback for urgent warnings. Frames and sessions are not persisted.

Building with Codex

Codex was the primary builder and product-design collaborator for SightLine.

In one continuous project task, Codex helped:

• Translate the initial PRD into a focused mobile product
• Design and implement the camera, voice, and verification loops
• Integrate GPT-5.6 vision and structured outputs
• Build the independent background safety watcher
• Add screen-reader semantics, large-text support, and reduced motion
• Test the API against real images and correct uncertainty handling
• Find and fix a mobile camera-mount timing issue
• Research the official competition rules and correct our submission track
• Create the interactive showcase, documentation, testing plan, demo script, and submission media

The repository includes a dated CODEX_LOG.md documenting this collaboration.

Challenges we faced

Making the product more than an image describer

The largest product challenge was ensuring SightLine demonstrated continuous reasoning rather than producing a richer caption. We solved this by making every task turn responsible for verifying the previous action before advancing.

Proactive safety without disrupting the task

Safety checks must not accidentally change the user’s current instruction. We separated normal task turns from background safety turns. A safety turn can interrupt for a hazard, but otherwise leaves the task unchanged.

Latency and cognitive load

Sending continuous video would be slow and expensive. SightLine instead sends resized, on-demand frames and produces one short instruction at a time.

Honest uncertainty

In a safety-adjacent product, pretending to know is worse than admitting uncertainty. Confidence is visible and audible, and SightLine asks for a clearer camera angle when the evidence is insufficient.

Accomplishments we are proud of

• A working perceive → reason → guide → verify loop
• Independent proactive hazard monitoring
• Hands-free task commands
• Explicit step verification and scene memory
• A complete screen-reader-friendly mobile experience
• A deterministic 45-second showcase requiring no camera or account
• Live GPT-5.6 responses that correctly lower confidence instead of inventing details

What we learned

We learned that accessibility is not a layer added after implementation. It changes the product architecture.

Short responses, predictable controls, honest confidence, large touch targets, minimal visual clutter, and conservative interruption behavior are central to SightLine—not optional polish.

We also learned that the strongest use of multimodal reasoning is not always describing more. Sometimes it is remembering what the user is trying to accomplish and deciding whether it is safe and appropriate to continue.

What’s next

Future versions could add:

• Personal task routines and preferences
• Optional escalation to a trusted human
• User-controlled safety sensitivity
• Realtime speech-to-speech interaction
• Wearable-camera support
• Evaluation with blind and low-vision testers

SightLine is assistive guidance, not a replacement for a cane, guide dog, or personal judgment. It is not intended for street crossing, driving, emergencies, or other safety-critical navigation.

Other AIs tell you what is there. SightLine helps you do the thing.

Built With

• api
• app
• codex
• gpt-5.6
• mediadevices
• next.js
• openai
• progressive
• react
• responses
• speech
• typescript
• vercel
• web
• zod