EchoSight AI

A real-time audio-spatial copilot using thermal computer vision and 3D acoustic panning to guide first responders safely through zero-visibility fire environments.

Comment

Inspiration

During heavy fires or natural disasters, first responders struggle with zero-visibility environments filled with smoke and debris. Traditional equipment forces rescuers to look down at thermal handheld monitors, taking their eyes off surrounding hazards. I was inspired to create EchoSight AI to turn spatial visual data directly into 3D directional sound, giving emergency responders an invisible acoustic co-pilot so they can navigate hands-free and save lives safely.

What it does

EchoSight AI is a real-time audio-spatial copilot for search and rescue operations. It takes video and thermal streams, identifies critical objects using edge computer vision, and translates spatial coordinates directly into 3D audio feedback:

  • Survivor Pinpointing: Plays a high-frequency panned audio ping ($880\text{ Hz}$) that moves in stereo sound depending on where a human heat signature is detected.
  • Hazard Avoidance: Generates a low-frequency pulsing warning ($180\text{ Hz}$) when structural collapse risks or toxic flares are spotted.
  • Voice Alerts: Provides hands-free synthesized voice updates and a 360-degree tactical radar interface for off-site incident commanders.

How I built it

I built EchoSight AI using a modern web and audio architecture:

  • Frontend HUD & Interface: Built with Next.js 15 (App Router), TypeScript, and Tailwind CSS using a dark tactical emergency HUD theme.
  • Computer Vision Engine: Implemented using HTML5 Canvas API frame processing and object detection algorithms to track spatial target positions ($X, Y, Z$).
  • 3D Audio Engine: Developed with the native browser Web Audio API (AudioContext, StereoPannerNode, and OscillatorNode) to pan spatial frequencies across left and right audio channels based on target alignment ($X = -1.0$ to $+1.0$).
  • Voice Synthesis: Integrated the Web Speech API for instant hands-free voice warnings.

Challenges I ran into

  • Low-Latency Audio Panning: Smoothing out real-time stereo panning without causing audio popping or delay when targets move rapidly across the video canvas.
  • Visual-to-Acoustic Translation: Finding the exact balance of sound frequencies and pulse rhythms so that responders can intuitively distinguish between human heat signatures and structural collapse hazards without getting overwhelmed by sound.

Accomplishments that I'm proud of

  • Successfully engineering a seamless spatial audio engine purely with native browser APIs without relying on heavy external audio libraries.
  • Designing a live interactive simulation mode where clicking on the HUD or running animated video paths dynamically drives the Web Audio spatial panner in real time.
  • Building a complete, working tactical dashboard prototype with an emergency UI from scratch.

What I learned

  • Deepened my knowledge of the native Web Audio API, specifically using StereoPannerNode and dynamic oscillator controls for real-time spatial positioning.
  • Mastered responsive canvas rendering and bounding box calculation techniques in Next.js 15.
  • Understood how to design low-friction hands-free user experiences tailored specifically for critical real-world emergencies.

What's next for EchoSight AI

  • Hardware Integration: Integrating physical thermal camera sensors and bone-conduction headset hardware for field testing.
  • Edge ML Optimization: Fine-tuning custom YOLOv8 thermal vision models running on local edge devices to function entirely offline.
  • Multi-Responder Mesh Network: Expanding the Incident Commander dashboard into a multi-agent map showing live spatial tracking for full response teams operating simultaneously.

Built With

Share this project:

Updates

Submission history