yeti-fc_SignBridge

The glasses in action on a live demo!
The 3D-prined ASL glasses, equipped with a camera on its left side.
A glimpse of the software in use. We trace hands with MediaPipe and give a confidence rating with our model.

SignBridge: Bridging the Deaf Communication Gap

About the Project

Inspiration:

We came up with our idea after an unfortunate realization: for many in the Deaf and hard-of-hearing community, a simple doctor’s visit, team meeting, or interview can be a source of immense anxiety. We were deeply moved by the story of Zulma “Yary” Santiago, an ASL instructor who was once refused treatment for a serious case of bronchitis because her medical provider didn't have an interpreter available.

When patients are forced to choose between bringing a third-party translator (which can compromise medical privacy) or facing a total communication breakdown, the system has failed. We wanted to build a tool that empowers individuals to communicate autonomously and privately, ensuring that no one is turned away from seeking care or conversing due to a language barrier.

How We Built It:

We aimed to create a wearable, non-intrusive prototype using 3D printing and accessible hardware. We chose the Raspberry Pi as our central processor due to its compact size and our existing proficiency in Python.

The technical architecture consists of a custom Computer Vision pipeline:

MediaPipe Hands: We utilized MediaPipe for real-time hand landmark detection. It allowed us to track 3D coordinates of the hand with high precision and low latency, which is critical for the fluid nature of sign language.

TensorFlow: We implemented a TensorFlow model to interpret the coordinate data and map it to specific ASL signs.

Custom Data Collection: To ensure the device worked reliably in a variety of settings, we created our own dataset. We found that general ASL datasets didn't account for the specific camera angles of a wearable device or the lighting typical of a clinic. By recording our own data, we tailored the model to the exact environment of the prototype.

Challenges & Hurdles

Building a functional wearable presented several hurdles:

Hardware Integration: Fitting a camera, Raspberry Pi, and onto a person required constant iteration. We faced issues with weight balance, requiring several redesigns of the glasses.

The Soldering Iron: Miniaturizing the electronics meant soldering in very tight spaces. Dealing with loose connections and ensuring a stable power supply for the camera module was a significant time investment.

Letters or Words: A major hurdle was the logic of the conversation itself. Initially, our model struggled to differentiate between when a user was fingerspelling a specific name or term (letter by letter) and when they were using a single gesture for a full word. We had to build a specific feature to toggle between these modes. Without this distinction, the system would try to force a "word" interpretation on a single letter, leading to total gibberish in a medical context.

What We Learned

This project taught us that engineering is most impactful when it addresses human dignity. We learned how to integrate complex machine learning models into constrained hardware environments, but more importantly, we learned that technology should serve as a tool for independence.