EchoVision

Introduction Page
Home Page

Inspiration

Close your eyes for ten seconds. Now try to pour yourself a glass of water.

That is everyday reality for 2.2 billion people living with vision impairment worldwide. The smartphone in their pocket has a 48-megapixel camera that can see everything they can't. The hardware exists. The voice assistants exist. Yet nobody wired them together hands-free.

We didn't set out to build another generic "AI for good" project. We set out because of a specific moment: watching a visually impaired user struggle to hold a medicine bottle at arm's length trying to read tiny dosage print. Their phone had 5G and voice assistants, yet it was completely inaccessible in that moment.

EchoVision was born from a simple, stubborn question: What if you could just ask a voice assistant "What's in front of me?" and instantly hear a natural, spatial answer in your native language? Not a robotic label, but a real human sentence: "There is a white medicine bottle on your desk, about six inches to the left of your keyboard."

What it does

EchoVision turns any web browser + Alexa into a real-time, hands-free visual assistant for the visually impaired:

Instant Web Access: No app installs, no App Store downloads. Open a single URL on any phone or laptop, and the camera starts streaming.
Alexa+ & Voice Integration: Say "Alexa, ask smart vision what is in front of me" or tap the mic on the web app, and within 2 seconds, she describes your surroundings.
True Multilingual Intelligence: Native Hindi (देवनागरी लिपि), Spanish, French, and English. Scenes and spatial relationships are synthesized from scratch in the target language with zero translation lag.
2-Tier Cost & Latency Triage: Reading text on a document or label is handled locally via edge OCR at $0.00 cost. Complex spatial scene understanding routes dynamically to NVIDIA NIM multimodal VLM.
Empathy Perspective Filter: Includes an interactive visual acuity simulation allowing sighted caregivers and developers to experience varying degrees of macular degeneration and low vision.

How we built it

We designed backward from Alexa’s strict 8-second webhook timeout and real-world network constraints:

Frontend: Vanilla HTML5, CSS3, and WebGL with zero heavy framework overhead. Camera video streams via native navigator.mediaDevices.getUserMedia WebRTC.
Backend Architecture: High-throughput Python FastAPI server deployed on Render with an in-memory frame buffer, eliminating database disk I/O bottlenecks.
Model Context Protocol (MCP) Server: Implemented a self-hosted Streamable HTTP MCP server (spec 2025-11-25) exposing perceive_scene for Alexa+ and autonomous agent orchestration.
Cloud Brain: NVIDIA NIM microservices running Meta’s LLaMA-3.2-11B-Vision-Instruct, optimized at the native $448 \times 448$ single-tile ViT resolution for sub-second Time-To-First-Token ($< 1.0\text{s}$).
Automated Keep-Alive: GitHub Actions cron automation running every 10 minutes to maintain persistent 24/7 backend availability without consuming GPU credits.

Challenges we ran into

The Multi-Tile Queue Trap: When capturing frames at 720p, the vision transformer split images into 4 separate tiles, causing cloud queue latency to exceed 25 seconds. By profiling the model's native patch architecture, we locked the background canvas to 448px (1 native tile), dropping generation latency to under 2 seconds while preserving text sharpness.
The Hindi Transliteration Trap: Standard LLM prompts frequently return Romanized Hindi ("Aapke samne..."). We engineered prompt constraints enforcing pure Devanagari script generation with custom end-of-sentence punctuation (।).
Beating Alexa's 8-Second Wall: Initial roundtrips took 8.3s. We streamlined the pipeline by using in-memory frame sync, SSE token streaming, and an instant non-streaming fallback buffer to guarantee sub-2-second responses.

Accomplishments that we're proud of

Sub-2-Second End-to-End Latency: Real-time multimodal vision inference streaming back from cloud infrastructure to user audio in under 2 seconds.
Zero-Install Universal Accessibility: Operates seamlessly on low-cost $50 smartphones and modern laptops alike with zero platform lock-in.
40% Cost Elimination: Edge OCR triage routes text-only queries locally, preserving expensive GPU compute.
100% Open Source: Fully licensed under the MIT License with public GitHub Actions workflows and self-hosted MCP endpoints.

What we learned

Accessibility is an architectural discipline: Fast latency, fallback buffers, and hands-free voice UX must be designed into the foundational protocol, not patched on at the end.
Strict constraints breed better systems: Alexa's 8-second cutoff forced us to eliminate every wasteful disk write, package bloat, and excess token pass.

What's next for EchoVision

Spatial Binaural Audio: Positioning audio descriptions in stereo 3D sound so users hear where an obstacle is located relative to their ears.
Smart Glasses Hardware: Prototyping lightweight ESP32-CAM and wearable frame integrations to eliminate holding a phone.
On-Device WebAssembly Models: Exploring fully offline browser-based quantized vision captioning for areas with zero connectivity.