Inspiration
Deaf and hard-of-hearing people miss sounds that carry real information — a knock, a doorbell, a delivery. Those moments don't need amplification, they need to be visible. We built awareness, not surveillance: a display that says "possible knocking — check the door" without pretending to know direction, distance, or cause. Raw microphone audio stays on the machine, and nothing leaves it unless the user explicitly opts in.
What it does
Live Sound Radar listens through one microphone, classifies sound locally, and turns knocks and doorbells into cautious visual cards: listening status, the latest detection with a model score, history, acknowledgement, and separate privacy controls. It never claims certainty or emergency — it says "Possible knocking" and shows a score, not a probability. It keeps detecting offline. Spoken alerts, cloud metadata, and AI summaries are optional, independently controlled, and off by default.
How we built it
A C++20 capture process (miniaudio, cpp-httplib) uses a preallocated SPSC ring and a worker to resample to 16 kHz mono, encode one-second s16le chunks, and send them over loopback HTTP with sequence, timestamp, RMS, and drop headers. A supervisor drives a heartbeat/lease protocol so capture only runs when authorized and stops within two seconds if the backend vanishes. A FastAPI + TensorFlow YAMNet service keeps a two-second window with a one-second hop, applies latch/cooldown rules, and stores canonical events in SQLite. The React/TypeScript/Vite dashboard polls a cursor-based state API. Snowflake export uses a parameterized, idempotent MERGE plus real AI_COMPLETE; speech maps fixed templates through ElevenLabs.
Challenges we ran into
Realtime boundaries came first: no allocation, locks, or I/O in the audio callback, and bounded memory so a stalled receiver drops chunks instead of stalling capture. Coordinating three components meant freezing schemas early, and strict header/sequence validation made mismatches obvious. The environment fought back — TensorFlow had no wheel for Python 3.14 (installed 3.12 and pinned a lockfile), Windows throttled the freshly built unsigned capture binary, and our first live knock tests scored zero because the mic was muted and later because AirPods voice processing flattened the transient. Switching to the laptop mic took knock scores from 0.00 to 0.42–0.99.
Accomplishments that we're proud of
The local path is real and verified: a live knock produced a genuine YAMNet event (model_score 0.834, ~20 ms per two-second window) and rendered in the actual dashboard, with capture running 24/24 chunks and zero drops. Capture passes lifecycle, fixture, and stall scenarios, including bounded memory under a slow receiver. We have honest failure states — disconnected, model loading, degraded capture — not fake detections. Raw audio never leaves the laptop, and the project ships with 66 Python and 24 frontend tests plus SQLite history that survives restarts.
What we learned
Contracts before code: writing the audio framing, event schema, and state cursor down first let three workstreams move in parallel and made integration bugs mechanical instead of mysterious. Validate physical assumptions early — the model was never the problem, the microphone was. Accessibility changes engineering decisions: visual-first alerts, keyboard acknowledgement, no color-only or sound-only cues, and speech as a supplement rather than the alert. And honest labeling is a feature: a model score beats "74% certain," both as truth and as product design.
What's next for AcousticIntelligence
Complete the optional integrations end-to-end: live ElevenLabs generation with real credentials, the cloud host with MongoDB Atlas persistence, the Snowflake export and AI summary scheduler, and the retention/deletion flow across both stores. Then tune detection on more real sounds and background noise to hit our demo targets (≥8/10 visible detections, no nuisance alerts), add a third class only if it earns its place, and rehearse the demo including offline and provider-outage paths. Longer term: sign and package the capture binary, ship a one-command launcher, and publish a placement/threshold guide so it works in a real room, not just at our desk.
Log in or sign up for Devpost to join the conversation.