Missing Whisper

Inspiration

India is home to over 63 million people who are deaf or hard of hearing, yet Indian Sign Language (ISL) has almost no digital infrastructure compared to ASL or BSL — no real-time translators, no accessible video-calling tools, barely any structured datasets. Deaf individuals routinely depend on a scarce, expensive human interpreter just to attend a doctor's appointment or a job interview. We kept coming back to one question: what if the device someone already owns could be the interpreter? That question became MissingWhisper.

What it does

MissingWhisper is a real-time, bidirectional ISL ⇄ speech communication platform with four modes:

Real-Time Communication — live two-way ISL ⇄ speech video calls over WebRTC, with MediaPipe-based sign detection and speech translation running live.
Standalone Translator — a sign → text/speech and speech → text studio for quick translation without a call.
3D Avatar & Signify Engine — a signing avatar for fingerspelling and animated sign playback.
Group Communication — multi-participant rooms with active-speaker detection and live captions.

We also extended it onto Amazon Alexa: a custom Alexa Skill routes voice intents through AVS into our existing API, reusing our Whisper-style STT and TTS-Indic pipeline so Echo devices become a hands-free entry point, and an Echo Show can render the 3D avatar or captions as an ambient interpreter.

Accessibility is foundational, not bolted on — WCAG 2.1 AA compliance, full keyboard navigation, reduced-motion support, and high-contrast modes are built in from the first line of code.

How we built it

Monorepo (npm workspaces): apps/api (Express + Socket.io signaling), apps/web (React + Vite + Redux), apps/isl-engine (PyTorch sign recognition), apps/signify (3D Babylon.js avatar viewer), and packages/shared-types as the single compiled source of truth for every contract across the stack.
Real-time layer: WebRTC for audio/video, Socket.io for signaling, gesture, and transcript relays.
CV & DL: MediaPipe hand/pose landmarks feeding a two-stream transformer sign-recognition model.
Speech: Whisper-style STT and an Indic TTS service for natural speech output.
Voice front-end: an Alexa Skill on AVS, authenticated via JWT, calling the same REST/typed contracts the web app uses.
Resilience by design: Postgres/Redis sit behind interface abstractions with in-memory fallback drivers, so the app runs with zero infrastructure locally but refuses fallback in production.
Infra: Docker Compose for local Postgres/Redis, Terraform for cloud provisioning, Vercel/Render for deployment.

Challenges we ran into

Bundle size vs. 3D avatar fidelity — our Babylon.js avatar chunk alone gzips to ~1.35 MB. We isolated it into a lazy-loaded route and enforced a hard 150 KB budget on the main entry bundle via a custom CI script.
Typed contracts across five runtimes — keeping REST, Socket.io events, and the Python ISL engine in sync required compiling shared-types to a real dist/ package instead of relying on source-only TypeScript types, after it quietly broke production builds.
Low-connectivity reliability — real-time sign recognition and WebRTC calls assume decent bandwidth, which isn't guaranteed everywhere in India, so we had to design graceful degradation instead of hard failures.
Making a voice-first device useful to a visual-language user — Alexa is audio-first by nature, so the integration only became meaningful once we used Echo Show's screen to surface the avatar and captions, not just the mic.

Accomplishments that we're proud of

A fully working bidirectional pipeline — not just sign-to-text, but speech-to-sign too — across video calls, a standalone studio, and group rooms.
Shipping WCAG 2.1 AA accessibility as a default, not a retrofit, across the entire web app.
Getting a heavy 3D avatar engine into a production-realistic, budget-enforced bundle without sacrificing visual fidelity.
Extending the same backend, unmodified, to power a completely new client — Alexa — proving the architecture's contracts were genuinely reusable rather than web-specific.
A backend that runs with zero infrastructure locally (in-memory fallbacks) while still being production-honest about what's degraded.

What we learned

Accessibility tooling fails most often not from missing features but from being an afterthought bolted onto an existing system. Building WCAG compliance, graceful degradation, and typed cross-service contracts in from day one meant every later feature — including the Alexa integration — was a matter of reusing solid foundations rather than retrofitting them. We also learned how differently "accessibility" reads depending on the device: what makes a web app accessible to a deaf user is almost the inverse of what makes a voice assistant accessible to one.

What's next for Missing Whisper

Expand the ISL vocabulary and dataset — our recognition model currently covers a limited gesture set; the biggest unlock is more labeled ISL data.
On-device inference — move sign recognition closer to the edge to cut latency and work in low-bandwidth regions.
Deeper Alexa skill certification — pursue official Alexa Skills Store listing and explore Alexa Built-in device partnerships for always-on ambient interpretation.
Interpreter marketplace fallback — for signs the model can't confidently translate, route to a human interpreter on demand instead of failing silently.
Mobile apps — native iOS/Android clients so MissingWhisper travels with users beyond the desktop and smart-speaker contexts.