SignBridge

Inspiration

SignBridge began with a question: How can we make everyday communication easier when one person uses American Sign Language and the other does not?

A question in a classroom, a conversation with a coworker, or a simple request can become difficult when people do not share a language. We wanted to explore how the camera already in someone’s pocket could help bridge that gap.

Our vision is a world where the way someone communicates never limits their opportunity to connect. SignBridge is our first step toward that vision.

What it does

SignBridge is a mobile prototype that turns supported ASL signs into readable, editable text.

Users start the camera, perform a sign, and review three possible word matches. They choose the intended word and can edit, copy, or clear the resulting message. The interface keeps the camera and text prominent, with Dictionary available when needed.

The app includes 500 built-in words, along with personally taught gesture patterns. If recognition struggles with a user’s gesture, they can choose its meaning, record three examples, and save a personal version. Completed personal words also appear in the searchable Dictionary.

Recognition runs locally on the phone. Personal examples are saved locally only when the user explicitly chooses to save them.

How we built it

We built the mobile experience using React Native, Expo, and TypeScript, supporting iOS and Android through a shared codebase.

Behind the camera, Google’s MediaPipe tracks landmarks on the hands, face, and upper body. Each hand has 21 tracked points. Following these coordinates across frames gives the app a numerical representation of movement over time.

Pretrained models associated with ASL Citizen and WLASL analyze sequences of selected landmarks. We run these models directly on the device using TensorFlow Lite. The recognition pipeline also checks a mirrored view and uses a secondary model when generating suggestions.

Personalization follows a separate process. We save examples under the meaning selected by the user, then compare fresh gestures with those examples using dynamic time warping. This helps accommodate differences in signing speed without retraining the entire neural network.

Challenges we ran into

Our biggest challenge was discovering how easily similar movements can produce different meanings.

During testing, signs sometimes returned unrelated words, while others produced no useful result. These failures pushed us to investigate the entire pipeline: camera framing, landmark visibility, sign boundaries, movement timing, and suggestion ranking.

Personal teaching introduced another challenge. A user could complete a recording, tap Save, and still see no progress because an overly strict tracking-gap check rejected the example. We revised both saving and personal matching to handle brief pauses while continuing to reject recordings with substantial missing data.

We also learned that adding features could quickly clutter the interface. We moved Dictionary and advanced tools out of the main workflow so users could focus on signing, reviewing, and communicating.

Accomplishments that we're proud of

We connected live camera tracking, local AI inference, editable text, and personal gesture learning in one working mobile prototype.

We are especially proud of making personalization part of the experience. Users can retain multiple gesture versions for a word, restore saved examples, and find completed personal words in Dictionary.

We also built correction into the main interaction. Users review suggestions and control the final message. Automated regression checks help us preserve saving, navigation, recognition behavior, and camera continuity as the product evolves.

What we learned

We learned that tracking movement and understanding language are different challenges. Landmarks provide useful information, but recognition also depends on training data, temporal patterns, and preprocessing that matches the models’ expectations.

We learned that restricting vocabulary does not automatically create high accuracy. Software tests can verify behavior, but real-world signing performance requires independent evaluation with representative users.

We also learned to treat failed recognition as valuable feedback. Each failure helped us identify a more specific question to investigate—and reminded us why users need clear correction controls.

What's next for SignBridge

Our next priority is reliable evaluation with consenting ASL users across different phones, lighting conditions, signing speeds, and signing styles.

We want to improve recognition of similar signs, measure incorrect suggestions, and determine how much personalization helps on fresh recordings.

We also plan to develop our experimental sentence translation feature and explore text-to-speech and speech-to-text for two-way exchanges. Supporting additional sign languages would require dedicated data, linguistic expertise, and evaluation.

Throughout that work, we want Deaf community members to help shape the vocabulary, interactions, and priorities. Our ambition is to turn this prototype into a useful communication tool grounded in the experiences of the people it serves.