SignBridge

Inspiration

Communication gaps in classrooms are still a real problem for deaf and hard-of-hearing students, especially in environments where expensive assistive hardware or polished mobile apps aren’t practical. I wanted to explore whether a simple, low-cost setup — just a laptop and a webcam — could still support meaningful sign language recognition using machine learning. SignBridge started as an experiment to understand how far classical ML models and smart system design could go before needing heavy deep-learning pipelines or specialized devices.

What it does

SignBridge is a dual-mode sign language classifier that recognizes both static hand poses and motion-based gestures in real time.

• Static signs: A, H, GOOD, BAD
• Motion signs: HELLO, J, Z

It uses MediaPipe to extract hand landmarks from a webcam feed, then applies:

• a static model for single-frame poses, and
• a motion model for 16-frame gesture sequences.

A confidence-based gating system decides which model’s prediction to trust, prioritizing accuracy and stability over speed. The result is a live, low-latency demo that runs on low-end hardware without a GPU.

How I built it

I built SignBridge end-to-end as a solo project:

• Used MediaPipe HandLandmarker to extract 21 hand landmarks (x, y, z)

• Converted landmarks into feature vectors:

  • 63 features for static signs
  • 1008 features (16 frames × 63) for motion gestures
  • Trained two RandomForest classifiers:
  • One for static classification
  • One for motion sequence classification

• Implemented a motion gate using wrist displacement to activate motion inference only when meaningful movement is detected

• Optimized performance using frame skipping, reduced resolution, MJPG capture, and landmark caching

Everything from dataset collection to live inference runs locally on a standard Windows laptop.

Challenges I ran into

• Motion detection reliability: Differentiating intentional gestures from natural hand jitter required careful threshold tuning.
• Low-end hardware constraints: Running MediaPipe and ML inference together initially caused severe FPS drops.
• Dataset limitations: All data was collected by a single signer, making overfitting a constant concern.
• Prediction flicker: Without confidence gating and cooldown logic, outputs changed too rapidly to be usable.

Each issue pushed me to focus more on system design and tradeoffs rather than just model accuracy.

Accomplishments that I am proud of

• Building a complete pipeline: data collection → training → evaluation → live demo
• Achieving real-time performance on non-GPU hardware
• Designing a clean dual-model architecture instead of forcing one model to do everything
• Shipping a working, explainable ML system instead of a mockup or concept
• Completing the entire project solo within a short hackathon timeframe

What I learned

This project taught me that:

• Smart architecture can matter more than complex models
• Classical ML still has real value when used thoughtfully
• Performance optimization is just as important as accuracy
• Clear problem framing makes technical decisions much easier
• Building something usable is harder — and more rewarding — than just training a model

What's next for SignBridge

Future improvements include:

• Expanding the vocabulary and adding more motion gestures
• Training with multiple signers for better generalization
• Porting the system to Android
• Exploring lightweight hardware integration (ESP32 / edge devices)
• Adding sentence-level interpretation instead of isolated signs

SignBridge is currently a technical MVP, but it demonstrates a realistic foundation for accessible, low-cost sign language tools.

Built With

• mediapipe
• numpy
• opencv
• python