EchoSight

Turn any camera into active eyes for the blind: continuous real-time path guidance, halting collisions mid-stride, and finding lost items with 100% voice control.

Comment

Inspiration

Over 2.2 billion people live with visual impairment worldwide. While AI has made incredible strides, the daily reality for a blind person walking across a room or navigating their home remains fraught with physical danger.

Existing assistive apps suffer from two major flaws:

  1. The Touchscreen Trap: They force blind users to tap flat glass screens to find buttons and dismiss modals.
  2. The Latency Hazard: Generic vision tools take 4 to 6 seconds to recite verbose descriptions ("A well-lit room with modern decor..."). When walking blind, latency equals collision. A blind person doesn't need poetry—they need immediate guidance: "Stop! Wall directly ahead."

We built EchoSight for the Alexa+ Track: an entirely hands-free, voice-first spatial co-pilot that acts as active eyes for the blind, protecting every footstep and restoring spatial independence.

What it does

EchoSight turns your phone camera into a real-time sensory radar operated entirely through natural voice:

  • Continuous Real-Time Walking & Path Guidance: Watches your path step-by-step and provides proactive directional cues ("Hold on. Stop right there, wall directly ahead", "Doorway on your right, step through carefully").
  • Instant Collision Prevention: Halts collisions mid-stride before impact happens.
  • Sub-50ms Walking Clearance: When a user asks "Can I move forward?", EchoSight verifies its live spatial safety state and answers in under 50 milliseconds.
  • 3D Spatial Memory Registry: Users can casually pin personal items ("Remember where my keys are"). EchoSight anchors objects relative to room landmarks and guides users back to them on demand.
  • 100% Hands-Free Interface: From startup greeting ("Say Enter to begin") to live path scanning, object recall, and teardown ("Stop"), zero touchscreen interaction is required.
  • Conversational Barge-In: Users can interrupt ongoing voice guidance mid-sentence the moment an obstacle appears or a command is spoken.

How we built it

  • Model Context Protocol (MCP): EchoSight implements the open MCP specification (2025-11-25+), exposing 9 specialized spatial and vision tools over Streamable HTTP/SSE (Server-Sent Events) and Stdio, allowing it to serve as the physical eyes and spatial cortex for Alexa+.
  • Frontend Cockpit: Built with React 19 and TypeScript in Strict Mode, bundled with Vite 8, and styled with Tailwind CSS. It incorporates native Web Speech API (SpeechRecognition & SpeechSynthesis) with continuous listening and acoustic echo suppression.
  • Backend MCP Server: Node.js and Express powering an SSE transport layer with hard timeout deadlines and fail-safe safety normalization.
  • Vision Intelligence: High-speed multimodal spatial vision engine paired with an intelligent multi-key rotation pool and rolling rate-budget tracker to sustain an adaptive 2.5-second radar cadence.

Challenges we ran into

  1. Eliminating Latency for Walking Safety: Standard multimodal cloud calls often take several seconds, which is dangerous when walking. We engineered a dual-tier safety loop and sub-50ms local safety cache to clear footsteps immediately while deeper scans run asynchronously.
  2. True Hands-Free Audio Handshake: Browsers restrict audio autoplay without user gestures. We created an auditory speech handshake where the system announces itself and listens for a single spoken word ("Enter") to unlock full two-way audio.
  3. Conversational Barge-In Without Feedback Loops: Preventing the microphone from picking up EchoSight's own voice speech synthesis required precise acoustic suppression states and millisecond-level synthesis cancellation when user speech is detected.
  4. Spatial Object Grounding: Teaching the AI to remember an object relative to permanent room structures (e.g., "two feet to the right of the desktop monitor") required strict JSON schema normalization in our MCP spatial registry.

Accomplishments that we're proud of

  • Created an assistive AI where a visually impaired person literally never has to touch the smartphone screen.
  • Achieved sub-second obstacle reflexes and sub-50ms clearance answers, turning a smartphone into an active white cane companion.
  • Built a fully standard-compliant Model Context Protocol (MCP) implementation that bridges modern voice assistants (Alexa+) with live physical vision.
  • Designed an experience that treats users with dignity—delivering crisp, practical walking instructions instead of verbose poetic descriptions.

What we learned

  • Speed is safety: When designing for the blind, every fraction of a second saved in sensory processing directly prevents physical injury.
  • Voice-first requires unlearning GUI patterns: Designing with zero buttons forced us to rethink auditory cues, conversational pacing, and fail-safe fallbacks.
  • The power of MCP: Standardizing vision tools over Model Context Protocol opens up immediate modularity for next-generation proactive voice assistants like Alexa+.

What's next for EchoSight

  • Smart Glasses Integration: Porting EchoSight's client layer to smart glasses (Ray-Ban Meta, Echo Frames) for completely unobtrusive, chest/eye-level camera alignment.
  • Haptic Feedback: Pairing voice alerts with spatial directional vibrations for noisy outdoor environments.
  • Outdoor & Transit Navigation: Expanding the spatial memory registry to recognize crosswalk signals, bus stop numbers, and curb drops.

Built With

Share this project:

Updates

Submission history