EchoVision 3D

Giving sight through sound: EchoVision turns video streams into 3D binaural audio cues and AI narration to guide blind users safely around obstacles.

Comment

Inspiration

Visually impaired individuals face constant navigation challenges in unfamiliar or dynamic environments. Traditional tools like white canes detect ground-level physical obstacles directly in front of the user, but they lack spatial awareness for head-height hazards, far-field obstacles, or identifying what objects actually are.

We were inspired to build EchoVision to bridge this gap by converting visual spatial data into intuitive 3D soundscapes. Just as bats navigate through echolocation, EchoVision translates real-time camera imagery into binaural audio cues, letting users "hear" the structure, distance, and identity of their surroundings in real time.

What it does

EchoVision is an assistive computer vision system that captures real-time video streams and translates spatial environments into 3D directional sound and natural spoken feedback.

  • 3D Directional Panning: Positional audio shifts seamlessly across left-to-right stereo channels using Web Audio API HRTF panners to reflect object location.
  • Pitch-Based Elevation: High, medium, and low pitch frequencies communicate object height (e.g., head-height obstacles vs. ground-level steps).
  • Proximity & Velocity Feedback: Proximity triggers louder, faster pulsing audio cues, while moving objects generate Doppler-style pitch shifts and Time-to-Collision (TTC) alerts.
  • Tiered Detection Pipeline:
    • Tier 1 (Classical CV): Always-on, model-free edge density (Sobel) and optical flow running locally with zero latency.
    • Tier 2 (Object Detection): Real-time 80-class object identification powered by YOLOv8n.
    • Tier 3 (Scene Narration): On-demand natural language scene narration via Groq Vision API (Llama 4 Scout).

How we built it

  • Frontend: Built as a zero-dependency client using HTML5, CSS3, Vanilla JavaScript, Web Audio API (HRTF PannerNodes, GainNodes, Oscillators), and the Web Speech API for TTS and voice commands.
  • Backend Engine: Built using Python and FastAPI, managing real-time full-duplex video frame streaming and audio metadata sync over persistent WebSockets at ~10 fps.
  • Computer Vision & AI: Combined OpenCV (Sobel operators, Lucas-Kanade optical flow, frame differencing) with PyTorch running lightweight YOLOv8 Nano models.
  • Cloud Vision Processing: Integrated Groq's low-latency Vision API running Llama 4 Scout for detailed multi-sentence scene descriptions.

Challenges we ran into

  • Web Audio Latency & Overload: Generating multiple continuous audio oscillators per detected frame initially caused browser audio crackling and thread blocking. We resolved this by implementing audio node pooling and gain smoothing.
  • WebSocket Heartbeat Disconnects: Heavy PyTorch inference passes on the CPU would occasionally block the main backend event loop, timing out the WebSocket. We solved this by offloading model execution to asynchronous thread pools and pre-warming PyTorch models on startup.
  • False Alarm Mitigation: Raw edge detection generated significant background noise in textured environments (e.g., patterned carpets). We engineered dynamic background calibration and baseline threshold filtering to ignore static background clutter.

Accomplishments that we're proud of

  • Zero-Latency Feel: Achieved real-time acoustic feedback (~10 FPS frame loop over WebSockets) with sub-50ms audio spatialization response times.
  • Model-Free Fallback: Built an active optical flow and edge-density detection engine that functions entirely offline without requiring AI weights or internet connectivity.
  • Fully Web-Native UI: Designed a single-file web client requiring zero native app installation, making it instantly accessible across smartphones and laptops.

What we learned

  • How to effectively control spatial sound field rendering using Web Audio API HRTF (Head-Related Transfer Function) panners.
  • Methods for balancing real-time computer vision performance against edge server compute constraints.
  • Efficient async patterns in FastAPI for handling binary image frame streams alongside JSON metadata payloads over WebSockets.

What's next for EchoVision 3D

  • Stereo Depth Sensing: Integrating dual-camera inputs or WebXR Depth APIs to replace monocular depth estimations with true physical depth maps.
  • Wearable Hardware Integration: Packaging the frontend onto lightweight Smart Glasses paired with bone-conduction headphones for clear environmental audio.
  • Custom Edge Fine-Tuning: Training specialized ultra-lightweight vision models specifically targeted at indoor accessibility hazard identification (e.g., wet floors, drop-offs, low hanging branches).

Built With

Share this project:

Updates

Submission history