Inspiration
Los Angeles is one of the most linguistically diverse cities on Earth, yet a Deaf person can still struggle with something as simple as asking a pharmacist about a prescription, checking in at an urgent care, or disputing a charge at a bank counter. Interpreters are essential, but they are expensive, often need to be booked days ahead, and are rarely available for a two-minute interaction at a corner store.
We wanted a tool for those everyday moments: a phone propped on a counter that lets a signer and a hearing person talk to each other in real time. It is not a replacement for certified interpreters. It is a bridge for the gaps between them and an ordinary person.
What it does
What it does
ASLTalk is a progressive web app that works in both directions:
- Sign to speech: The phone camera watches the signer and ASLTalk recognizes ASL signs. It shows live captions and speaks the sentence aloud in a natural voice.
- Speech to sign: The hearing person speaks. ASLTalk transcribes it and shows captions alongside looping sign demonstrations for key phrases.
- Context modes: Pharmacy, Hospital, Bank, and Shop modes load vocabulary and phrasing for that setting (for example "pain," "medicine," "account," "how much").
- Confirm before speaking: In high-stakes modes, the signer sees the sentence ASLTalk is about to speak and can approve or redo it, so a misread never goes unchecked. ## How we built it 1. Capturing the hands. MediaPipe extracts 21 landmarks per hand, plus face and pose landmarks, from each video frame, entirely in the browser. Each frame becomes a feature vector with 2 hands $\times$ 21 landmarks $\times$ 3 coordinates $= 126$ values.
2. Normalizing. To make recognition independent of camera distance and position, we translate each landmark relative to the wrist and scale by hand size:
$$\hat{p}_i = \frac{p_i - p_0}{\lVert p_9 - p_0 \rVert}$$
where $p_0$ is the wrist and $p_9$ is the base of the middle finger.
3. Recognizing signs. A sequence of $T$ normalized frames goes into a lightweight sequence model (a [1D-CNN + BiLSTM / small Transformer]) trained on a sign vocabulary from public ASL datasets ([WLASL / ASL Citizen]) plus clips we recorded ourselves. Training minimizes categorical cross-entropy:
$$\mathcal{L} = -\sum_{k=1}^{K} y_k \log \hat{y}_k$$
We only accept a prediction when its softmax confidence clears a threshold $\tau$ and stays stable over several frames:
$$\max_k \hat{y}_k \ge \tau$$
4. Turning signs into sentences. ASL grammar differs from English. A stream of
recognized glosses like STORE ME GO WHERE is passed to the Claude API with the current
context mode, which returns a natural English sentence ("Where is the store?"). The prompt
is constrained to the recognized glosses, so the model does not invent content.
5. Speaking and listening. Text-to-speech produces the voice output. Speech-to-text handles the hearing person's reply, which is displayed as captions and matched to our sign-clip library.
6. Keeping it fast. Real-time conversation needs a tight latency budget:
$$t_{total} = t_{capture} + t_{landmarks} + t_{classify} + t_{LLM} + t_{TTS}$$
7. Privacy by design. Video never leaves the phone. Only landmark coordinates are processed, and nothing is stored.
Challenges we ran into
- Sign language is more than hands. Facial expression, posture, and movement carry grammar. We added face and pose landmarks, but we are honest that this is still an approximation.
- Knowing where one sign ends and the next begins. Continuous signing has no clear word boundaries. We used a motion-based segmentation window and a short pause rule.
- Lighting, skin tones, and backgrounds. Early models failed in dim rooms and with busy backgrounds. We added augmentation (brightness, rotation, scale) .
- Small, imperfect datasets. Public datasets cover limited vocabulary and signers, so we scoped to [N] high-value everyday signs rather than pretending to translate all of ASL.
- Latency vs. accuracy. Bigger models were more accurate but too slow on a phone. We traded model size for responsiveness and added the confirm step to protect accuracy.
- Trust in high-stakes settings. A wrong word at a hospital matters. That led to the confirm-before-speaking design and a clear "not a substitute for a certified interpreter" notice.
Accomplishments that we're proud of
- A working two-way demo running in a mobile browser with no install
- Privacy-first design where no video leaves the device
What we learned
- Accessibility tools must be designed with the community, not just for it.
- Landmark-based approaches are lighter, faster, and more private than raw-video models.
- Honest scoping beats overclaiming. A reliable 50 signs is more useful than an unreliable 5,000.
- Pairing a small recognition model with a language model handles the grammar gap between ASL and English better than word-by-word translation.
What's next for ASLTalk
- Expand vocabulary and add fingerspelling for names and numbers
- Add more regional and cultural variation in signing
- Pilot with pharmacies, clinics, and bank branches around Los Angeles