Hum: names the sound, hints at the meaning, runs on your device
Some people communicate with sounds, not words. A new caregiver starts from nothing.
Every number measured on data the model never saw
Two layers: what kind of sound (measured) and what it means for this person (learned from family)
One person's sounds, coloured by the meanings their family gave
Sound type: 90.3% correct on people and recordings it never heard
A confident answer, and the caregiver's own label agrees
A cough from a stranger: named correctly, never interpreted as a message
Each meaning: what helps, example sounds, and how far to trust Hum
Meaning scores, with the inflated number you would get by mixing recording sessions
Hands-free listening: each sound picked out, a daily log, and an alert when several sound upset
A passport for the next caregiver, and a small voice file to hand over to their phone
Works on any phone, and offline after one visit
When torn, Hum offers two meanings plus what this kind of sound meant for this person before
Upset alert reliability for each voice: dependable for some, flagged as not for others

Inspiration

Some people communicate with sounds instead of words: many autistic people, and people with cerebral palsy or certain genetic conditions, are minimally speaking. Their families learn what each sound means. Everyone else starts from nothing: a new teacher, a substitute nurse, a babysitter. Requests go unanswered and distress gets read as misbehaviour.

I wanted to carry a family's knowledge to everyone else, on an ordinary phone, without ever uploading anyone's voice.

What it does

Hum listens and answers in two layers.

1. What kind of sound it is (objective). A model names 11 kinds of sound: laugh, cry, scream, cough, sneeze, sniff, sigh, throat clearing, yawn, breathing, or other voice. It is 90.3% correct on 4,703 clips from people and recordings it never heard. Body sounds like a cough are named, never interpreted.

2. What it likely means for this person (personal). Hum learns each person's sounds from their caregivers and answers with a set of meanings: one when it is sure, two when it is torn, "maybe" when it is not. It never bluffs.

Around those two layers:

Teach with one tap. The personal model updates instantly, on the device.
Keep listening. Hands-free mode picks out each sound, keeps a daily log by sound type (exportable for a doctor or therapist), and raises an upset alert when several sounds in a row lean upset.
Patterns. "Past cries from Voice 16 meant frustrated 45 of 84 times."
Passport and hand-off file. A printable page for a new caregiver, and a 4 KB voice file that moves a taught voice to another phone. The file holds numbers, not recordings.
Works offline. Everything runs in the browser. No audio leaves the device.
An evidence page inside the app shows every score, including the ones that failed.

How I built it

Data. ReCANVo (MIT, 2023): 7,077 real vocalizations from 8 minimally speaking people, labelled in the moment by their families. For sound types: VocalSound, Nonspeech7k and EmoGator.
Models. Two 1.2M-parameter CNNs in PyTorch. The log-mel front end is built from fixed convolutions, so each ONNX file (5.7 MB) takes raw 16 kHz audio and the browser output matches Python to four decimals. The meaning encoder was pretrained on 32,130 vocal bursts (EmoGator), trained with cross-entropy plus a supervised contrastive loss, and uses a prototype per meaning, so teaching is just adding a vector.
Honest answers. Split conformal prediction per person. Given calibration scores \( s_i = 1 - \hat p(y_i \mid x_i) \), Hum offers every meaning whose probability clears \( 1 - \hat q \), where \( \hat q \) is the \( \lceil (n+1)(1-\alpha) \rceil / n \) quantile, so the true meaning lands in the set about \( 1-\alpha \) of the time.
App. Static HTML, CSS and JavaScript with onnxruntime-web, the Web Audio API (an AudioWorklet segments live sound), and a service worker for offline use. Hosted on GitHub Pages; no backend.

Results, measured honestly

	Score
Sound type, held-out people and recordings	90.3% accuracy, macro-F1 0.87 (91.4% on VocalSound vs the published 90.6%)
Meaning, 5-fold cross-validation, sessions kept apart	macro-F1 0.34 vs 0.13 for always guessing the commonest meaning; right meaning in the top two 69% of the time
Upset alert (replayed over every session)	catches about 7 in 10 upset sounds with under 5% false alarms for 3 of 8 voices; not dependable for the rest, and the app says so

Challenges

Leakage. Random train/test splits put clips from the same recording on both sides. The same model scores 0.47 that way and 0.30 when sessions are kept apart. I rebuilt every evaluation around whole sessions.
A hard ceiling on meaning. I tried 11 ideas to raise meaning accuracy: four pretrained speech models (wav2vec 2.0, DistilHuBERT, Whisper, voc2vec), fine-tuning, EmoGator pretraining, ensembles, pitch and rhythm features, session context, active learning. None beat the shipped model beyond noise. The MIT authors found the same: a caregiver's interpretation uses context a microphone cannot hear.
So I changed the question. Instead of over-promising meaning, I added a layer that can be accurate (what kind of sound) and made the meaning layer honest about its uncertainty.

What I learned

How to evaluate properly: session-held-out splits, cross-validation, bootstrap intervals, and calibration.
That a negative result is worth reporting.
That on-device ML in a browser is practical.
That "useful and honest" beats "impressive but wrong", especially for vulnerable users.

What's next

Test with families, teachers, and speech-language pathologists before any real use.
Measure the sound-type layer on minimally speaking people. Their recordings have no type labels yet, so on them it is unmeasured.
More voices. Eight is too few to claim generality.

Built on ReCANVo (Johnson, Narain, Quatieri, Maes, Picard, 2023). Inspired by the MIT Media Lab Commalla project; not affiliated. Hum is a hint for caregivers. It does not diagnose, and the people who know the communicator are always right.