SignSpeak

Inspiration

430 million people are deaf or hard of hearing. When they want to give a presentation or a TED Talk, they have no voice of their own — they rely on interpreters who put someone else's words in their mouth. We wanted to change that.

What it does

SignSpeak converts ASL signs into emotionally expressive speech in real time. A deaf presenter signs naturally, their facial expressions are detected to determine emotion, and a voice speaks their words with matching tone — happiness, sadness, anger, surprise — so they sound like themselves, not a robot.

How we built it

React frontend, MediaPipe for hand gesture and facial landmark detection, a custom emotion classifier using 52 facial blendshapes, a custom-designed gesture recognition system trained on the specific signs used in our demo with an architecture designed to scale to the full ASL vocabulary given more training time, a sentence cleaner for contextual correction, and Google TTS for low-latency emotional speech output.

Challenges we ran into

Running hand detection and facial emotion detection simultaneously caused MediaPipe WASM conflicts. Per-face calibration was essential because blendshape thresholds vary significantly between individuals. TTS latency started at 5+ seconds.

Accomplishments that we're proud of

A fully working end-to-end pipeline built in 24 hours. All vision processing runs on-device — no video ever leaves the user's machine. The emotion modulation genuinely sounds different across emotions, not just different labels.

What we learned

Facial expressions carry as much meaning in ASL as hand signs — ignoring the face is why most ASL tools fall short. Scoping ruthlessly is a feature. Streaming APIs are a completely different mental model from request-response.

What's next for SignSpeak

Replace custom-designed demo gestures with a fully trained model covering the complete ASL vocabulary — the pipeline is already built for it. Full continuous sentence-level signing and a bidirectional bridge so hearing people can respond in spoken language.