something like good

Inspiration

What it does

Inspiration

My grandmother lost most of her hearing in her seventies. Watching her nod along to family conversations she couldn't actually follow — smiling at jokes she never heard — was the quiet heartbreak that started this project. Live captioning exists, but it's built for lecture halls and TV broadcasts, not for a noisy dinner table with five people talking over each other. We wanted to build Murmur: a real-time, speaker-aware captioning system that runs entirely on-device, so conversations stay private and captions stay fast.

What it does

Murmur listens to multi-speaker conversations and renders a live, color-coded transcript on a phone or tablet — each speaker gets their own lane, like a group chat happening in real time. It performs:

• Streaming speech-to-text with sub-400ms latency, fully on-device
• Speaker diarization ("who said what") using lightweight voice embeddings
• Semantic smoothing — an LLM pass that repairs disfluencies ("uh, the thing, you know, the—the report") into readable text without changing meaning
• Zero cloud dependency — nothing ever leaves the device

How we built it

The pipeline is a three-stage stream: audio → diarized text → smoothed captions.

1. Capture & VAD. We chunk microphone input into 30ms frames and run voice-activity detection to gate the pipeline, cutting idle compute by ~70%.
2. Diarization. Each speech segment is embedded into a 192-dimensional speaker vector. We cluster incrementally with an online variant of agglomerative clustering, assigning a segment to speaker $k$ when the cosine similarity exceeds a threshold:

$$k = \arg\max_i ; \frac{\mathbf{e} \cdot \mathbf{c}_i}{\lVert \mathbf{e} \rVert , \lVert \mathbf{c}_i \rVert}, \quad \text{accept if} ; \geq \tau = 0.72$$

 where $\mathbf{e}$ is the segment embedding and $\mathbf{c}_i$ is the running centroid for speaker $i$. New speakers spawn a new centroid automatically — no enrollment step.

1. Transcription & smoothing. A quantized Whisper-small model (int8, ~240MB) handles STT, and a distilled 1B-parameter language model rewrites each finalized utterance for readability. We run both on the phone's NPU via ONNX Runtime.
2. UI. A React Native front end renders the caption lanes with optimistic partial results — words appear grayed-out while tentative, then solidify as the decoder commits.

Challenges we ran into

• Latency vs. accuracy is a brutal trade-off. Whisper wants long context windows; live captioning wants instant output. Our fix was a two-tier decode: emit greedy partial hypotheses immediately, then silently replace them with beam-searched finals ~1.5s later. Users perceive it as "captions that self-correct," which testers actually liked.
• Diarization collapse. With similar-sounding speakers (two brothers in our test group), centroids merged and the transcript became one giant monologue. We added a temporal prior — penalizing rapid speaker-switching within 300ms — which cut confusion errors by 41%.
• Thermal throttling. Twenty minutes in, the phone got hot and the NPU downclocked, blowing our latency budget. We ended up duty-cycling the smoothing model: it only runs on finalized utterances, never on partials.

Accomplishments that we're proud of

• End-to-end latency of 380ms median on a mid-range Android phone, fully offline
• Diarization error rate of 8.2% on our 4-speaker dinner-table test set, down from 23% in our first prototype
• Our first real user test: my grandmother followed a full 40-minute family conversation and, for the first time in years, interrupted someone to disagree. That was the whole point.

What we learned

We learned that "real-time" is a user perception, not a benchmark number — self-correcting captions with 380ms partials feel faster than perfect captions at 900ms. We also learned to profile on real hardware early: everything that worked on our laptops fell apart under mobile thermal constraints, and half our engineering time went into fitting the pipeline inside a power budget, not improving accuracy.

What's next for Murmur

• Multilingual code-switching — our family mixes Korean and English mid-sentence, and current models handle that poorly
• Directional audio using multi-mic beamforming to physically separate speakers before diarization
• Wearable target — porting the pipeline to smart-glasses SDKs so captions appear in your field of view, not on a table ## How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for something like good

Built With

• codex