Signbridge

Bridging the communication gap with real-time ASL translation and a compassionate AI friend.

Comment 3

Inspiration

We live in an age of remarkable technological capability and yet loneliness is rising, accessibility gaps persist, and countless people fall through the cracks of systems that were never designed with them in mind. For millions of deaf and hard-of-hearing individuals, navigating a world that wasn't built for them can be incredibly isolating. Inspired by the Oracle Challenge, our mission was to use AI and machine learning to close the gap between human need and human connection. We built SignBridge: not just a tool for translation, but an autonomous, empathetic AI companion designed to actively listen and respond with compassion.

What it does

SignBridge is a completely private, zero-latency web application that operates in two modes:

Live Translator: Using only a standard webcam, it translates real-time ASL fingerspelling (A-Z) and common ASL phrases (like "Hello", "I Love You", "A Little") into English text. It supports dynamic velocity gestures, allowing users to literally "swipe" their hand across the screen to delete a word. The resulting text is then spoken aloud to deploy natural, human-sounding audio with ElevenLabs. AI Buddy: An integrated chat companion powered by Google Gemini to understand language like a human and build a chatbot that gives personalized advice. The Buddy isn't just a generic chatbot; it is programmed with a strict "Safety Protocol." If the user signs anything indicating emotional distress or isolation, the AI autonomously shifts into a highly compassionate validation mode to support them. It even knows local Tampa geography to suggest accessible, highly visual activities if the user signs that they are bored.

How we built it

We completely avoided heavy, laggy cloud-based vision models Computer Vision: We implemented Google's MediaPipe Hand Landmarker locally via WebAssembly. This generates a 3D hand mesh with 21 coordinates. Because it runs 100% locally on the device, it ensures absolute privacy for the user. The Heuristic Engine: Instead of a black-box machine learning model, Ibuilt a custom deterministic heuristic engine in JavaScript. By calculating the 3D Euclidean distance and joint angles of the PIP, DIP, and MCP joints in real-time, we can accurately classify ASL signs instantly. The AI & Audio: I integrated the Gemini 2.5 Flash API for the Buddy's brain to analyze input and respond empathetically , and the ElevenLabs API to create realistic, dynamic, and emotionally expressive voices for our companion.

Challenges we ran into

Building a velocity-tracking gesture system in a browser was incredibly difficult. Initially, when a user would swipe their hand horizontally to trigger our "swipe to delete" command, the transitional frames of the hand moving would accidentally trigger static signs like "Hello" or "I Love You." We solved this by implementing a dual-solution: We engineered a "Stability Check" that tracks the history of the wrist's X-coordinates, disabling static word recognition if the hand is actively moving. We built a sleek UI toggle to give the user explicit control between "A-Z Spelling Mode" and "Full Words Mode," completely eliminating false positives and improving the user experience.

Accomplishments that we're proud of

I'm proud of building a fully functional, zero-latency computer vision application entirely in the browser in under 24 hours. More importantly, we are proud of the AI Buddy's safety protocol. Seeing the AI seamlessly detect a distressed input and pivot into an empathetic, human-centered response proved that we successfully built technology in service of humanity.

What we learned

We learned the immense difference between simple machine-learning API wrappers and building custom mathematical logic (using Euclidean distances and dot-products) for deterministic computer vision. Also We didn't work with MediaPipe before, We was only aware of it's existence. But through this project, We learned MediaPipe and I'm confident I'll be able to use it in the future as well.

What's next for Signbridge

We want to expand the AI Buddy into a two-way learning tool. Our next step is implementing a reverse-lookup dictionary with 3D animated sign models, allowing hearing users to type English into the chat and have the Buddy physically demonstrate how to sign the response in

Built With

Share this project:

Updates