Signal

Inspiration

Our inspiration came from the film CODA, which follows Ruby, the only hearing member of a Deaf family, who becomes her parents' and brother's voice in nearly every conversation with the hearing world, from the docks of their fishing business to the doctor's office. Ruby loves her family deeply, yet her role as interpreter slowly becomes a weight that holds her back from her own dreams; and her family, in turn, is left depending on her to be understood. One of the film's most powerful moments comes at Ruby's choir concert, when the sound drops away and the audience experiences the performance as her family does: a room full of people moved by something they cannot hear. That silence stayed with us.

CODA made us realize that the burden of communication too often falls on the people closest to the Deaf community, or on Deaf people themselves. Interpreters are not always available, especially in the small, everyday moments: ordering coffee, asking for directions, catching a last-minute gate change at the airport. Meanwhile, most sign language apps ask the signer to prop up a phone and perform for a camera that sees them from the outside, as though they were being observed rather than heard. We asked a simple question: what if the technology saw signing the way the signer does? This is not merely a question of convenience, but of independence: no one should have to wait for a Ruby to be in the room in order to be heard.

What it does

Signal is a first-person, two-way ASL communication system. Rather than pointing a camera at the signer, Signal is worn by the signer, capturing their hands from their own point of view through smart glasses and a chest-mounted camera.

Sign to voice: Signal recognizes signing and, instead of stringing isolated words together, interprets full signed thoughts into natural English sentences. Those sentences are then spoken aloud in a lifelike voice, so the hearing person hears a conversation, not a list of vocabulary.
Speech to sight: When a hearing person replies, Signal listens through the glasses' microphones and displays their words as captions directly in the user's line of sight. The user never has to look down at a phone and lose eye contact.
Announcements, captured: Public announcements, such as a pilot or gate agent speaking over a crowded terminal, are some of the most easily missed moments for Deaf travellers. Signal transcribes them onto the heads-up display, so vital information no longer passes them by.
Reading the nod: Not all communication happens through the hands. Using the glasses' motion sensors, Signal detects head nods and treats them as a natural "yes," letting conversations flow the way they do in real life.

Together, these features turn a one-way toll road into a two-way street, where both people can speak and both people can understand.

How we built it

Signal is built from four layers that work together: capture, understanding, voice, and display.

Capture: We paired Rokid Glasses, which serve as both a head-mounted camera and a heads-up display, with a vertical, wide-view chest-mounted camera. This gives us two egocentric angles on the signer's hands: one that sees what the signer sees, and one that keeps the hands in frame when they drift below the glasses' field of view. The glasses stream video over Wi-Fi Direct to a phone, which handles the heavy processing.
Data: Nearly every public ASL dataset films the signer from the front, which is the opposite of our perspective. We therefore labeled our own first-person footage so our recognition model learns what signs look like from the inside out.
Understanding: Recognized signs pass through a context-listening layer and a grammar model. ASL has its own grammar, word order, and use of space, so this layer reorganizes signed input into proper English sentences rather than translating word for word.
Voice: The finished sentences are sent to the ElevenLabs API, which gives Signal a natural, expressive voice rather than a robotic one.
Head motion: The glasses' onboard sensors track head movement, and we built detection for nods as a conversational response.
Display: Incoming speech and announcements are transcribed and rendered as captions on the glasses' HUD, positioned to stay readable without blocking the user's view.

We also leaned heavily on AI-assisted development with Claude Code to move quickly across the many integrations a 24-hour build demands.

Challenges we ran into

The greatest obstacle was not the code, but access. Rokid's device binding is restricted to enterprise developers, which meant we could not simply plug in and build. Under the pressure of a 24-hour clock, we reached out to Rokid directly and explored the AIUI Studio route as an alternative path onto the hardware.

Additionally, the first-person view is a double-edged sword. It is exactly what makes Signal unique, yet it means hands drift in and out of frame, overlap one another, and appear at angles no public dataset accounts for. This is why we added the chest-mounted camera, and why we had to build and label our own data from scratch rather than borrowing someone else's.

Moreover, ASL is not English spoken with the hands. It carries its own grammar, facial expressions, and spatial structure, and a sign's meaning can shift with context. Teaching a model to respect that structure, instead of producing sentences that are technically correct and humanly wrong, proved far more complex than recognizing individual signs.

Finally, keeping the experience feeling like a real conversation was a constant balancing act. Every step, from capture to recognition to grammar to voice, adds delay, and a pause that is too long breaks the natural rhythm of speaking with someone.

Accomplishments that we're proud of

We are proud that we built a system that sees signing from the signer's own perspective; an approach that challenges the default assumption that the signer must perform for a camera. Rather than adapting the person to the technology, we adapted the technology to the person.

We are proud that Signal works in both directions. Many translation tools only help the hearing person understand the Deaf person; Signal gives both people a voice and both people a way to understand.

We are proud of building our own first-person dataset under extreme time pressure, and of pushing past understanding single signs toward understanding full sentences with proper grammar.

We are proud of persevering through hardware restrictions that could have ended the project in its first hours, and of bringing together glasses, a chest camera, a phone, a grammar model, a voice engine, and a heads-up display into one experience within 24 hours.

Most of all, we are proud that Signal is being open sourced, so the work does not end when the hackathon does.

What we learned

We learned that translation is an act of interpretation, not substitution. A word-for-word system can produce sentences that are technically accurate yet entirely miss what the person meant, and real understanding requires context, grammar, and respect for ASL as a language in its own right.

We learned how much perspective matters, both in the camera and in the design process. Building for the signer, from the signer's own vantage point, changed every decision we made, from where we mounted the cameras to where captions appear on the display.

We learned how to work with wearable hardware, real-time video streaming, motion sensors, and voice APIs, and how quickly a plan must adapt when a key piece of access is locked behind an enterprise wall.

Lastly, we learned that accessibility technology should be built with the community it serves, not merely for it. The film that inspired us reminded us that the people best equipped to define good communication are the ones who navigate its barriers every day.

What's next for Signal

We are open sourcing Signal. Communication access should not sit behind a paywall or inside a single company's product; it should belong to the community it serves. By making our code, models, and first-person dataset openly available, we hope other developers, researchers, and, most importantly, members of the Deaf community will build on it, correct it, and push it further than we ever could in 24 hours.

From here, we plan to:

Expand the dataset with more signers, signing styles, lighting conditions, and environments, so Signal works for everyone, not only the people who built it.
Refine the grammar model to better capture facial expressions, spatial references, and the full richness of ASL.
Reduce latency so conversations feel as natural as speaking face to face.
Test with Deaf users and let their feedback shape every next step.
Grow beyond ASL to other sign languages, since there is no single universal sign language.

Where other tools build a wall between two worlds, we want Signal to build a doorway, one both people can walk through together.