About the Project

Inspiration

Millions of deaf and hard-of-hearing individuals struggle with communication barriers, especially in fast, everyday interactions where human interpreters are unavailable. We wanted to build a real-time, accessible, low-latency sign language translation system that works directly in a browser without requiring expensive hardware or cloud GPUs.

What it does

SignBridge enables users to sign in front of a camera, and the system performs the following functions: 1. Captures hand, pose, and face landmarks in real time 2. Extracts rich motion and spatial features across 32 frames 3. Predicts the gloss (base sign) using a lightweight transformer 4. Accumulates glosses into a sentence 5. Applies a grammar-correction NLP model to produce natural language 6. Outputs spoken English (or text) back to the user

The entire pipeline is built for speed, accuracy, and accessibility, enabling seamless, real-time sign-language communication.

How we built it

Frontend (React) • Used MediaPipe Tasks (HandLandmarker, PoseLandmarker, FaceLandmarker) for web-based landmark extraction • Implemented a 32-frame temporal buffer to capture each sign • Wrote a JavaScript mirror of our preprocessing pipeline (normalization, curl angles, palm normals, trajectory features) • Used ONNX Runtime Web to run our trained transformer model inside the browser • Displayed translated text and generated speech on the fly

Backend (Python + Django) • Designed a full-feature engineering pipeline, including: • torso normalization • hand extent normalization • finger curl computation • palm normal vectors • motion deltas • trajectory statistics • two-hand relational features • Trained a compact transformer-based gesture-to-gloss classifier using PyTorch • Exported the model to ONNX for deployment on the web • Used Django to serve the web app and coordinate NLP grammar correction • Deployed the entire system on Oracle Cloud

⚠ Challenges we ran into • Getting consistent landmarks across browsers, lighting conditions, and speeds • Designing a shared preprocessing pipeline that works identically in both Python and JavaScript • Achieving low-latency inference within 5–10 ms per gloss • Creating a dataset representation that includes motion, handshape, and relational features • Ensuring the grammar model could correctly reconstruct spoken English from glosses • Optimizing ONNX Runtime Web for smooth real-time performance

Accomplishments that we’re proud of • Built a fully real-time sign-language translation system in the browser • Designed a comprehensive, research-grade feature extraction pipeline • Achieved minimal latency while maintaining strong accuracy • Successfully synchronized preprocessing across Python and JavaScript • Deployed a complete end-to-end AI system on Oracle Cloud • Built a highly modular system that can support any sign language dataset

What we learned • Deep understanding of hand/pose landmark engineering • Advanced feature extraction for gesture motion & handshape • How to deploy ML models across Python, ONNX, and JavaScript • Real-time constraints for computer-vision pipelines • Importance of normalization consistency in multimodal AI systems • How gloss-based language grammar differs from spoken English

What’s next for SignBridge • Add continuous sign-sentence segmentation for fully fluid conversation • Expand sign-language coverage using custom datasets • Integrate multi-user or remote communication modes • Ship mobile app versions with on-device inference • Add support for avatar-based reverse translation • Explore few-shot learning for personal signing variations

Built With

Programming Languages: • Python • JavaScript • TypeScript

Frameworks & Libraries: • Django (backend) • React (frontend) • MediaPipe Tasks (Hand, Pose, Face Landmarkers) • PyTorch (model training) • ONNX Runtime Web (JS inference)

Cloud/Deployment: • Oracle Cloud (hosting + backend deployment)

Other Tools: • NumPy / SciPy (feature extraction) • ReportLab (PDF generation) • OpenCV (video/frame processing) • T5 / Transformer models (grammar correction)

Share this project:

Updates

Submission history