SoundSight

Inspiration

Sound carries critical situational context that visual cues alone often miss: emergency sirens, doorbells, footsteps, approaching vehicles, or sudden alarms. For deaf and hard-of-hearing individuals, navigating environments without these acoustic signals demands constant hyper-vigilance.

OFFGRID challenges builders not to build what they are supposed to build, but what they want to exist. We wanted an interface where acoustic reality isn't just transcribed as speech-to-text, but visually mapped as real-time spatial awareness—turning ambient soundscapes into clear, intuitive visual telemetry.

What it does

SoundSight is an assistive audio-visual scene analysis tool designed to bridge the auditory-visual gap. It listens to real-time ambient audio, detects and classifies critical acoustic events, and translates them into immediate visual indicators.

• Real-Time Sound Event Detection: Continuously classifies environmental sounds (e.g., sirens, alarms, glass breaking, door knocks, speech).
• Urgency & Severity Mapping: Color-codes and prioritizes critical acoustic hazards to immediately grab peripheral attention without overwhelming the user.
• Multimodal Visual Dashboard: Provides dynamic visual waveforms, confidence scores, and categorical alerts built for fast situational comprehension.

How we built it

• Frontend & Interface: Built using Streamlit for an interactive, reactive real-time telemetry dashboard.
• Audio Processing & Pipeline: Developed in Python, utilizing NumPy and audio streaming pipelines to capture and preprocess raw waveform signals.
• Acoustic Classification: Powered by multimodal/audio event recognition models and embeddings, matching real-time ambient spectrogram inputs against classified acoustic sound categories.
• Data Flow: Ambient Audio Capture $\rightarrow$ Fast Fourier Transform / Feature Extraction $\rightarrow$ Event Classification Inference $\rightarrow$ Dynamic Visual Alert Rendering.

Challenges we faced

• Latency vs. Accuracy: Processing streaming audio in near real-time without lagging the visual interface required fine-tuning audio chunk buffers and inference windows.
• Ambient Noise Filtering: Differentiating background white noise (HVAC, street rumble) from actionable foreground sound events (sirens, knocks) to minimize false positives.
• Designing Intuitive UI: Ensuring alerts are instantly interpretable at a glance without causing sensory overload.

Accomplishments that we're proud of

• Delivering a functional, responsive prototype that visualizes ambient sound with minimal perceptible delay.
• Building an impactful accessibility tool adhering strictly to the ethos of human-centric assistive tech.

What we learned

• How to manage real-time audio telemetry streams and buffer pipelines in Python.
• Practical constraints of acoustic event classification under varied background noise conditions.
• How thoughtful visual design and UX are just as vital as model accuracy in assistive technology.

What's next for SoundSight

• Wearable / Heads-Up Integration: Porting the visual alert layer to smart glasses (HUD) or wearable haptics (directional vibrations).
• Directional Sound Localization: Implementing multi-microphone array processing to show the exact physical direction (left, right, behind) a sound originates from.
• On-Device Edge Deployment: Compressing the model pipeline to run fully offline on edge devices without internet reliance.

Built With

• accessibility
• artificial-intelligence
• audio-processing
• machine-learning
• multimodal
• opencv
• python
• streamlit