SignBridge — Real-Time Sign Language Translation

Inspiration

Most software assumes everyone speaks. We wanted a webcam to turn sign language into English sentences in real time.

What it does

Webcam frames stream over WebSocket → MediaPipe detects hand landmarks → a classifier labels the gesture → confident signs are buffered and de-duplicated → Groq LLM (LangChain) turns them into a natural sentence, with a fallback if the LLM fails. Includes JWT/bcrypt auth and REST endpoints.

How we built it

Webcam → decode → hand detection → gesture classify → buffer → LLM → sentence

FastAPI + OpenCV + MediaPipe handle the real-time vision side; SQLAlchemy/SQLite back auth. The gesture classifier is a placeholder behind a clean Prediction(label, confidence) interface, so a trained model can drop in later without touching the rest of the pipeline.

Challenges

• Keeping detection real-time without letting slow LLM calls block the frame loop
• Smoothing noisy, flickering gesture predictions into clean words
• Designing solid interfaces for a classifier we hadn't trained yet
• Failing gracefully when the LLM is unavailable

What we learned

Structuring a multi-stage real-time ML pipeline with clean contracts, practical MediaPipe Tasks usage, and prompting LLMs for constrained, natural output.

What's next

Train a real classifier, stream sentences token-by-token, add conversation memory, support custom user signs.

Built With

• fastapi
• groq
• langchain
• mediapipe
• opencv
• python
• random-forest
• react
• sqlite
• tts