Inspiration
One in five people live with hearing loss, and everyday alert sounds - a baby crying, a knock at the door, a smoke alarm, a car horn - are invisible to them. Smart sound-alert gadgets exist, but they cost money, send audio to the cloud, or lock you into an app store. I wanted to see whether a real, trained neural network could run entirely inside a plain web page and turn any laptop or phone into a visual sound-awareness companion - no account, no server, nothing uploaded.
What it does
SoundCue listens continuously through the microphone. Every 2 seconds a small convolutional neural network classifies a 4-second audio window into 10 alert-relevant sound classes: baby crying, door knock, alarm clock, siren, car horn, glass breaking, dog barking, thunder, footsteps and running water. The whole pipeline runs locally in the browser.
- Live listen mode shows the top-3 class probabilities in real time, abstains when the model is unsure, and raises deduplicated, timestamped alerts on a visual timeline with three urgency tiers (the same sound within 10 seconds merges instead of spamming). High-tier alerts can flash the screen, and the log exports to CSV.
- Try a clip mode ships 10 real held-out test clips, accepts your own WAV uploads, and includes an interactive white-noise slider so you can watch confidence honestly degrade until the model abstains.
- Evidence page publishes measured results, per-class precision/recall and the confusion matrix.
Everything - FFT, mel filterbank, convolutions, batch-norm, the classifier - is re-implemented in pure JavaScript. The audio never leaves the device.
How I built it
- Data: ESC-50, 400 clips across the 10 chosen classes. Official folds 1-3 train (240), fold 4 validation (80), fold 5 test (80). No clip overlaps folds.
- Model: a 79,866-parameter CNN (4 conv+BN+ReLU+maxpool blocks, global average pooling, linear head) trained from scratch in PyTorch on CPU, seed 7, with speed/gain/noise/time-shift/mixup/SpecAugment-style augmentation.
- In-browser inference: the trained weights ship as base64 float32 in a JSON file (about 330 KB compressed);
dsp.jsre-implements FFT, the mel filterbank and every layer in vanilla JS, so inference needs no framework, no server and no API. - Frontend: plain HTML/CSS/JS with the Web Audio API, charcoal/lime high-contrast design, keyboard accessible.
Measured results (no vanity metrics)
All numbers were re-measured from the shipped weights with the final feature pipeline:
- CNN: 93.75% on the 80-clip held-out test split (validation 92.5%). A same-features baseline (mel mean/std + softmax regression) scores 56.2% test / 67.5% validation.
- Full parity test: the pure-JS implementation matches PyTorch on all 80 test clips - largest probability difference 3.8e-7, zero disagreements in predicted class or in the abstain decision.
- Test errors (5 clips): siren confused with car horn twice, dog with glass breaking, footsteps with door knock and with thunder.
- Honest caveat: the test split was inspected during development (interactive noise tests, pipeline fixes), so read it as a development-exposed benchmark, not pristine independent validation. No threshold or feature was retuned on it.
A feature I removed, on evidence
I built few-shot enrollment ("teach it your own doorbell": 3 samples, nearest-neighbour on the CNN's embeddings, distance gate) - and then measured it. Enrolling on 3 door-knock clips, the automatic gate false-fired on 36 of 72 non-knock held-out clips; even a gate tightened to half that distance still false-fired on 15 of 72. An awareness tool that cries wolf is worse than none, so the feature was cut. The evaluation script and results are in the repo.
Challenges
- Re-implementing a PyTorch CNN's exact numerics in JavaScript without a framework (the parity test was the hard, boring, essential part).
- Keeping the model small enough to download fast while staying accurate on 10 classes.
- Being honest in a hackathon: publishing the confusion matrix, the baseline, the removed feature and the provenance note instead of a shiny single number.
What I learned
Small CNNs on mel features are surprisingly strong when augmentation matches the deployment domain; shipping honest evidence is more valuable than another point of accuracy; and "it runs in the browser" is a real engineering constraint, not a demo trick.
What's next
Larger and more diverse training data (in-the-wild recordings, not just ESC-50), on-device adaptation that is evidence-gated rather than demo-gated, vibration/watch output, and evaluation with Deaf and hard-of-hearing users in real homes.
Disclosures
- AI assistance: used for implementation, evaluation and documentation under human direction; all reported numbers were re-measured from the shipped artifacts with the scripts in the repo.
- Licenses: app code MIT; ESC-50 dataset CC BY-NC 3.0; trained weights CC BY-NC 3.0 as a derivative; the 10 bundled demo clips keep their original per-clip licenses (7 CC0, 3 CC-BY, credits in
clip_attribution.json). Non-commercial use only, consistent with the dataset license. - Not a safety device: SoundCue is an awareness aid and research prototype, evaluated on 80 clean test clips - not in real homes. Do not rely on it for emergencies, smoke/CO alarms or medical monitoring.
Built With
- accessibility
- convolutional
- css3
- html5
- javascript
- machine-learning
- python
- pytorch