Inspiration

Roughly one in eight Americans has disabling hearing loss. The conversations that go wrong aren't appointments booked weeks ahead — they're the ninety seconds at a pharmacy counter or a clinic front desk. Too short to schedule an interpreter, too important to skip. So we asked a narrower question than "build an ASL translator": can a laptop the clinic already owns cover those ninety seconds, offline?

What it does

Signify turns a webcam into an on-device sign bridge. It recognises 250 isolated ASL signs (hello, thank you, water, pain, help…) and fingerspelled A–Z for names and medications. The signer confirms, the sentence builds, and it is spoken out loud. When the word model and the letter model both fire — a flat "hello" hand is the letter B — the app shows AMBIGUOUS and refuses to auto-commit. It never silently invents a sign.

How we built it

  • Recognition: MediaPipe Holistic landmarks → the upstream feature pipeline (30 fps resample, ≤64 frames, 460 dims) → ONNX. The word model is GestureSign (1.4M parameters, 5.6 MB). Fingerspelling is a CNN over a rendered skeleton, with motion rules for J and Z.
  • Decision layer: word scores are calibrated (T = 0.883), letter scores are raw softmax, so the router never compares them. Each model is judged only against its own threshold and freshness window, and only stable, unambiguous results are appended automatically.
  • Three front ends: an OpenCV desktop app, an upload API, and a browser client fed by a 15 Hz JSON state stream plus a 14 fps JPEG preview over WebSocket — so the page shows exactly the frames the models saw.

Challenges we ran into

  • MediaPipe 1.x removed the Holistic graph that produced the training landmarks, so we pinned 0.10.21 + numpy<2 + Python 3.11. The preprocessing contract, not the model, was the fragile part.
  • Both models on one CPU starved each other — ONNX Runtime's spin-wait blocked MediaPipe. Two threads with spinning disabled plus Hands on a 640 px copy took alphabet throughput from 13.5/s to 21.6/s, with landmarks mapped back to full-frame pixels so the upstream thresholds still meant the same thing.
  • Our first web client waited for five frames above 0.8 before accepting a label. It looked fine, because it was talking to a mock emitting random 0.85–0.99 scores. Against the real model that is unreachable: the same "hello" scores 0.42, 0.63 and 0.97, with a threshold of 0.29. We deleted the client logic and made the server authoritative.

Accomplishments that we're proud of

  • The full loop runs at ~15 updates/s on a laptop CPU — no GPU, no cloud, and the video never leaves the machine.
  • 75 tests running the real models, including parity against upstream's feature fixture at 1e-4 tolerance.
  • Limits we can state precisely: 72.1% top-1 for unseen signers is upstream's figure on 14,396 held-out clips; live-mode accuracy is written down as not measured.

What we learned

Accuracy was never the hard part; building an interface for uncertainty was — knowing which model may speak, when a result is stale, and when the honest answer is two candidates and a pause. Also, preprocessing is the model: most of our bugs were contract bugs, not modelling bugs.

What's next for Signify

  • Sentence-level — segmentation plus gloss-to-English over a small language model.
  • A vocabulary that fits the user, not a children's list.
  • Calibrated letters and per-signer few-shot adaptation.
  • Real user testing — we have zero Deaf users and zero interpreters in the loop.

Built With

Share this project:

Updates

Submission history