Inspiration
Moving about in the physical environment without sight is a very difficult task. Current technology such as white canes or screen reading software for the visually impaired can identify objects within reach of the user; however, they cannot provide location information and speed information of the object within the 3D environment. The system, EchoVision 3D, was designed specifically for this purpose by providing auditory information of visual depth, movement, and height information in an easy to understand 3D sound environment.
What it does
EchoVision 3D analyzes live video streams and converts environmental spatial information into binaural 3D audio cues:
3D Audio Spatialization: Uses Head-Related Transfer Functions (HRTF) to locate objects in terms of left/right pan, vertical position (pitch-encoded for ground/waist/head), and distance using volume gain/pulse.
Three Tiered Object Detection: Combines classical computer vision (optical flow & edge detection) for immediate local movement detection, YOLOv8n for real-time detection of 80 classes of objects, and Groq's Vision API (Llama 4 Scout) for on-demand spoken descriptions of the scene.
Collision Detection: Determines Time to Collision (TTC) based on optical flow vectors to produce Doppler pitch-shifting for approaching objects.
How we built it
Backend: Built with Python and FastAPI, managing full-duplex WebSockets for streaming video frames and spatial audio metadata.
Vision & AI Pipeline: OpenCV for classical edge filtering and optical flow; PyTorch with YOLOv8 Nano for local object detection; Groq's Vision API for low-latency scene descriptions.
Frontend & Audio Synthesis: Native JavaScript using the Web Audio API (PannerNode, GainNode, OscillatorNode) to synthesize spatial binaural audio in-browser without third-party audio engines.
Voice Interface: Integrated the browser Web Speech API for hands-free voice command control.
Challenges we ran into
Blocking the Event Loop: Execution of PyTorch passes on the same FastAPI loop was initially causing WebSocket timeouts. We solved that problem by processing AI inference asynchronously in separate threads.
Overload and Clipping of the Audio: Creating individual audio nodes for each detected bounding box was causing acoustic interference. We created a custom adaptive node pool/priority system based on hazard risk level.
Web Browser Auto-Play Policy Restrictions: Modern web browsers do not allow Web Audio API contexts to auto-play; instead, they require some sort of user interaction.
Accomplishments that we're proud of
Latency of less than 50ms achieved for audio and consistent video frame rate of 10FPS through WebSockets.
Developed a comprehensive 4D audio engine from scratch with native Web Audio API building blocks.
Maintained a perfect connection drop rate of 0% under heavy multi-tier AI inference loads.
What we learned
Encoding of multiple spatial dimensions of information (direction, distance, elevation, velocity) using only audio frequency and HRTF panning.
The method of striking the balance between extremely fast classical computer vision algorithm and computationally expensive inference from a deep-learning model.
What's next for EchoVision 3D
Porting Software to Wearable Devices: Embedding the software into wearable hardware configurations (such as smart glasses integrated with bone conduction headphones).
LiDAR/Depth Sensors Integration: Adding hardware LiDAR/depth sensors to enhance performance under low light conditions and indoors.
Edge Device Optimization without Cloud Connection: Using quantized local models that can operate solely on edge devices.
Log in or sign up for Devpost to join the conversation.