SignBridge

Real-time ASL gesture recognition in your browser. Breaking communication barriers for the Deaf community, one sign at a time.

Comment

Inspiration

Over 70 million people worldwide use sign language as their primary form of communication, yet most hearing people can't understand a single sign. This creates an invisible wall in everyday interactions — at hospitals, schools, and stores. We built SignBridge to start tearing that wall down.

What It Does

SignBridge is a real-time ASL gesture recognition web app. Open it in your browser, enable your webcam, and start signing. The app instantly translates your hand gestures into text displayed as a live chat transcript. It currently supports 6 ASL letters: A, B, C, L, W, and Y, with text-to-speech output so hearing people can also hear what's being signed.

How We Built It

  • Frontend: React + Framer Motion for animations, all in a single HTML file
  • Backend: FastAPI (Python) serving a REST /predict endpoint
  • AI Pipeline: Every second, the browser captures a webcam frame and POSTs it to the backend. MediaPipe Hand Landmarker detects 21 precise hand landmarks, which are then classified using finger position rules
  • CNN Model: A custom PyTorch CNN trained on the Sign Language MNIST dataset (27,455 training images) achieving about 90% test accuracy

Challenges We Faced

  • Camera conflict: The browser and Python can't share the webcam simultaneously. We solved this by having the frontend capture frames and POST them to the backend as images — Python never touches the camera directly.
  • Real-world accuracy: Models trained on clean dataset images don't generalize to messy webcam footage. MediaPipe's landmark-based approach solved this by working regardless of lighting or background.
  • Python 3.14 compatibility: Several libraries had silent failures requiring significant debugging.

What We Learned

  • How to build a full ML pipeline from dataset → training → deployed API
  • The gap between academic model accuracy and real-world performance
  • How MediaPipe hand landmark detection works under the hood
  • How to architect a real-time video processing pipeline

What's Next

  • Full ASL alphabet support
  • Word and sentence-level prediction using LLMs
  • Mobile support
  • Two-way communication (text-to-sign animation) ```

3. Built With:

Python, JavaScript, HTML, CSS, PyTorch, FastAPI, MediaPipe, 
React, Framer Motion, OpenCV, NumPy, Pandas, Uvicorn, 
Sign Language MNIST Dataset

Built With

Share this project:

Updates