SignBridge

Breaking communication barriers between the Deaf community and the hearing world, one gesture at a time.

Comment

Inspiration

Over 70 million people worldwide use sign language as their primary form of communication, yet most hearing people can't understand a single sign. This creates an invisible wall in everyday interactions — at hospitals, schools, stores, and beyond. We built SignBridge to start tearing that wall down.

What it does

SignBridge is a real-time ASL gesture recognition web app. You open it in your browser, enable your webcam, and start signing. The app instantly translates your hand gestures into text, displayed as a live chat transcript. It currently supports 6 ASL letters: A, B, C, L, W, and Y.

How we built it

  • Frontend: React + Framer Motion for smooth animations and a polished UI
  • Backend: FastAPI (Python) serving a REST endpoint for gesture prediction
  • AI/ML: MediaPipe Hand Landmarker detects 21 hand landmarks in real-time, which are then classified using rule-based finger position logic
  • CNN Model: A custom PyTorch CNN trained on the Sign Language MNIST dataset (27,455 training images) achieving 89.8% test accuracy
  • Pipeline: Browser captures webcam frames → sends to FastAPI → MediaPipe detects hand → model predicts gesture → result displayed as chat bubble

Challenges we faced

  • Camera conflict: The browser and Python couldn't share the webcam simultaneously. We solved this by having the frontend capture and send frames to the backend via HTTP, so Python never touches the camera directly.
  • Real-world accuracy: Models trained on clean dataset images don't generalize well to messy webcam footage. We solved this with MediaPipe's landmark-based approach which works regardless of lighting or background.
  • Python 3.14 compatibility: Several libraries had silent failures on the latest Python version, requiring significant debugging.

What we learned

  • How to build a full ML pipeline from dataset to deployed API
  • The gap between academic model accuracy and real-world performance
  • How MediaPipe's hand landmark detection works under the hood
  • How to architect a real-time video processing pipeline

What's next

  • Support for the full ASL alphabet
  • Word-level prediction (not just letters)
  • Mobile support
  • Two-way communication (text-to-sign animation) ```

Built With:

Python, PyTorch, FastAPI, MediaPipe, React, Framer Motion, 
OpenCV, Sign Language MNIST, HTML/CSS, JavaScript

Built With

Share this project:

Updates