Inspiration

Los Angeles is one of the most linguistically diverse cities on Earth, yet a Deaf person can still struggle with something as simple as asking a pharmacist about a prescription, checking in at an urgent care, or disputing a charge at a bank counter. Interpreters are essential, but they are expensive, often need to be booked days ahead, and are rarely available for a two-minute interaction at a corner store.

We wanted a tool for those everyday moments: a phone propped on a counter that lets a signer and a hearing person talk to each other in real time. It is not a replacement for certified interpreters. It is a bridge for the gaps between them and an ordinary person.

What it does

What it does

ASLTalk is a progressive web app that works in both directions:

  • Sign to speech: The phone camera watches the signer and ASLTalk recognizes ASL signs. It shows live captions and speaks the sentence aloud in a natural voice.
  • Speech to sign: The hearing person speaks. ASLTalk transcribes it and shows captions alongside looping sign demonstrations for key phrases.
  • Context modes: Pharmacy, Hospital, Bank, and Shop modes load vocabulary and phrasing for that setting (for example "pain," "medicine," "account," "how much").
  • Confirm before speaking: In high-stakes modes, the signer sees the sentence ASLTalk is about to speak and can approve or redo it, so a misread never goes unchecked. ## How we built it 1. Capturing the hands. MediaPipe extracts 21 landmarks per hand, plus face and pose landmarks, from each video frame, entirely in the browser. Each frame becomes a feature vector with 2 hands $\times$ 21 landmarks $\times$ 3 coordinates $= 126$ values.

2. Normalizing. To make recognition independent of camera distance and position, we translate each landmark relative to the wrist and scale by hand size:

$$\hat{p}_i = \frac{p_i - p_0}{\lVert p_9 - p_0 \rVert}$$

where $p_0$ is the wrist and $p_9$ is the base of the middle finger.

3. Recognizing signs. A sequence of $T$ normalized frames goes into a lightweight sequence model (a [1D-CNN + BiLSTM / small Transformer]) trained on a sign vocabulary from public ASL datasets ([WLASL / ASL Citizen]) plus clips we recorded ourselves. Training minimizes categorical cross-entropy:

$$\mathcal{L} = -\sum_{k=1}^{K} y_k \log \hat{y}_k$$

We only accept a prediction when its softmax confidence clears a threshold $\tau$ and stays stable over several frames:

$$\max_k \hat{y}_k \ge \tau$$

4. Turning signs into sentences. ASL grammar differs from English. A stream of recognized glosses like STORE ME GO WHERE is passed to the Claude API with the current context mode, which returns a natural English sentence ("Where is the store?"). The prompt is constrained to the recognized glosses, so the model does not invent content.

5. Speaking and listening. Text-to-speech produces the voice output. Speech-to-text handles the hearing person's reply, which is displayed as captions and matched to our sign-clip library.

6. Keeping it fast. Real-time conversation needs a tight latency budget:

$$t_{total} = t_{capture} + t_{landmarks} + t_{classify} + t_{LLM} + t_{TTS}$$

7. Privacy by design. Video never leaves the phone. Only landmark coordinates are processed, and nothing is stored.

Challenges we ran into

  • Sign language is more than hands. Facial expression, posture, and movement carry grammar. We added face and pose landmarks, but we are honest that this is still an approximation.
  • Knowing where one sign ends and the next begins. Continuous signing has no clear word boundaries. We used a motion-based segmentation window and a short pause rule.
  • Lighting, skin tones, and backgrounds. Early models failed in dim rooms and with busy backgrounds. We added augmentation (brightness, rotation, scale) .
  • Small, imperfect datasets. Public datasets cover limited vocabulary and signers, so we scoped to [N] high-value everyday signs rather than pretending to translate all of ASL.
  • Latency vs. accuracy. Bigger models were more accurate but too slow on a phone. We traded model size for responsiveness and added the confirm step to protect accuracy.
  • Trust in high-stakes settings. A wrong word at a hospital matters. That led to the confirm-before-speaking design and a clear "not a substitute for a certified interpreter" notice.

Accomplishments that we're proud of

  • A working two-way demo running in a mobile browser with no install
  • Privacy-first design where no video leaves the device

What we learned

  • Accessibility tools must be designed with the community, not just for it.
  • Landmark-based approaches are lighter, faster, and more private than raw-video models.
  • Honest scoping beats overclaiming. A reliable 50 signs is more useful than an unreliable 5,000.
  • Pairing a small recognition model with a language model handles the grammar gap between ASL and English better than word-by-word translation.

What's next for ASLTalk

  • Expand vocabulary and add fingerspelling for names and numbers
  • Add more regional and cultural variation in signing
  • Pilot with pharmacies, clinics, and bank branches around Los Angeles

Built With

  • browser-web-apis
  • camera-api
  • canvas-api
  • css
  • github
  • html
  • html5
  • javascript
  • json
  • local-storage
  • mediapipe-tasks-vision
  • mediapipe-webassembly-runtime
  • python
  • python-http-server
  • responsive-web-design
  • session-storage
  • speech-synthesis-api
  • tailwind-css
  • vibration-api
  • web-speech-recognition-api
Share this project:

Updates

Submission history