Auris

Inspiration

I was thinking about what AI is actually good at — not what sounds impressive, but what models are mature enough to be reliable. Audio event classification (YAMNet, trained on AudioSet's 521 sound classes) is one of those areas: it's been battle-tested, runs on-device, and doesn't hallucinate the way language models can.

Then I looked for who needs this most. Around 466 million people worldwide have disabling hearing loss (WHO). Many live alone. When they sleep, work with noise-cancelling headphones, or are simply in another room, they miss critical environmental sounds: a doorbell, a phone ringing, a smoke alarm, a baby crying, someone knocking or calling their name. Existing solutions are either expensive hardware devices with fixed generic sound libraries, or smartphone apps that can't recognize your specific doorbell.

The gap was clear: a free, browser-based tool that runs on any device, recognizes real environmental sounds (not just speech), and — crucially — can learn your*sounds, not generic ones. And since audio is deeply personal, it had to be on-device only. No uploading someone's home audio to a server.

What it does

Auris is a browser-based environmental sound awareness tool. Open the page on any device — phone, laptop, tablet — grant microphone permission, and it classifies sounds in real time using YAMNet (an on-device audio classification model).

• Priority-tiered alerts: Critical sounds (smoke alarm, siren, glass breaking) trigger a full-screen red overlay + vibration — impossible to miss. High-priority sounds (doorbell, phone, baby crying, alarm clock) show a prominent toast + short vibration. Normal sounds (knocking, dog barking) get a quiet toast.
• Personalized transfer learning: record your own doorbell or your name in 5 seconds. Auris extracts a 1024-dim audio embedding and learns to recognize your specific sound — not a generic one. Only the embedding vector is stored, never the audio.
• Calibrated decision layer: each sound class has its own confidence threshold (critical sounds use a low threshold so they're never missed), with multi-frame debouncing and cooldown to prevent flicker. When uncertain, it says so rather than guessing.
• Real-time spectrogram + top-5 readout for transparency — you can see what the model hears and what it's considering.
• Privacy-first: all inference runs on-device in the browser. No backend, no API keys, no audio upload. The app is a PWA that works offline after first load.
• Guided onboarding: a 5-step tutorial walks first-time users through the core features.

How we built it

• TensorFlow.js + YAMNet — the model is bundled locally (~15MB, 4 weight shards) rather than loaded from TensorFlow Hub, because the TF Hub URL started returning 403 after the Kaggle migration. This also means the app works fully offline after first load.
• Web Audio API + AudioWorklet — real-time audio capture with 16kHz resampling done inside the worklet thread (with a ScriptProcessorNode fallback for older browsers). The browser's native sample rate (often 48kHz) is linearly resampled to YAMNet's required 16kHz.
• Inference pipeline: each 0.975s audio window is fed through YAMNet, producing scores [N,521], embeddings [N,1024], and a spectrogram [N,64]. Scores are mean-aggregated across frames and softmax-normalized. Embeddings are mean-aggregated for the custom-sound matching path.
• Decision state machine: preset sounds are matched by substring-mapping AudioSet class names to our 9 target categories, each with an independent confidence threshold. Custom sounds are matched via cosine similarity to stored embedding vectors (threshold 0.70). Multi-frame debouncing (3 consecutive frames) + 5s cooldown prevent flicker. Critical sounds use a low threshold (0.20) — better to false-alarm than miss a smoke alarm.
• Vanilla JS + Vite — minimal, fast, static deployment. No framework overhead.
• PWA — manifest + service worker caches the app shell, model, and demo sounds for offline use.
• GitHub Pages + Actions — zero-cost hosting with automatic deploy on push.

Challenges we ran into

1. TF Hub model URL broke. The original tfhub.dev/google/tfjs-model/yamnet/tfjs/1 URL started returning 403 (TensorFlow Hub migrated to Kaggle). Fix: bundled the model files locally in /public/model/ — which turned out to be a feature, not a bug, since it enables full offline use.

2. Browser sample rate mismatch. YAMNet expects 16kHz mono, but browsers typically run AudioContext at 48kHz. Creating the context with { sampleRate: 16000 } is unreliable (some browsers ignore it). Fix: capture at native rate and resample ourselves with linear interpolation, done inside the AudioWorklet for the modern path and in the main thread for the fallback.

3. Inference API uncertainty. Some TF.js graph models only work with executeAsync, others with predict. Fix: try predict first, fall back to executeAsync on failure. Also added a warmup inference at load time that surfaces failures in the status badge instead of silently breaking later.

4. Demo reproducibility. Browser microphone capture depends on room noise and mic quality — unreliable for a demo. Fix: built a "Demo mode" that plays built-in sound files through speakers and feeds the decoded samples directly into the inference pipeline, bypassing the mic entirely. 100% reproducible.

5. GitHub Pages base path. Deploying to a sub-path (/auris/) broke all absolute fetch paths. Fix: used import.meta.env.BASE_URL for runtime fetches and scope-relative paths in the service worker.

Accomplishments that we're proud of

• Zero backend, zero API keys, zero audio upload. The entire app — model inference, transfer learning, alerts — runs in the browser as a static site. This is both a privacy win and a deployment win (free hosting, no scaling concerns).
• Personalized transfer learning in the browser. Recording a 5-second sample and having the model recognize your specific doorbell — using only embedding cosine similarity — is something commercial hardware products (which use fixed sound libraries) can't do.
• Calibrated confidence. The decision layer doesn't just pick the highest-scoring class — it uses per-class thresholds, debouncing, cooldown, and explicit "uncertain" labeling. A confident wrong answer is worse than no answer, especially for a safety tool.
• Accessibility-first design. Critical sounds trigger a full-screen overlay that's impossible to miss, plus vibration for mobile. The UI is dark, high-contrast, and works on phone screens.

What we learned

• Audio classification is a mature, reliable AI task — unlike LLMs, it doesn't hallucinate, runs fast on-device, and has clear confidence scores. This makes it well-suited for safety-critical applications where reliability matters.
• On-device ML is practical for production. TensorFlow.js + YAMNet runs real-time on a laptop browser with no noticeable lag. The model is ~15MB — a one-time download, then fully offline.
• Transfer learning doesn't require retraining. Extracting embeddings and doing cosine similarity is a simple, effective form of personalization that works with a frozen model — no GPU, no fine-tuning loop.
• Privacy can be a feature, not a constraint. "Audio never leaves your device" is both ethically important for this use case and technically simpler (no backend to build, host, or secure).

What's next for Auris

• Web Bluetooth wearable vibration pucks for sleep use — a deaf person sleeping can't see a screen, but a wrist vibration device could wake them for smoke alarms.
• Smart home integration — flash smart lights when a critical sound is detected (Philips Hue, Home Assistant via MQTT).
• Multiple custom sounds with incremental learning — recognize multiple household sounds simultaneously without retraining existing ones.
• Background-tab persistence — keep listening and alerting even when the browser tab isn't focused (Web Notifications + a persistent service worker).
• Multi-language alert text — currently English-only; the UI text should localize.
• On-device benchmarking — measure and display per-device inference latency so users know what to expect on their hardware.

Built With

• a11y
• accessibility
• audio-classification
• javascript
• machine-learning
• privacy
• pwa
• tensorflow
• tensorflowjs
• transfer-learning