Said With Hands

Inspiration

One of our close friends is deaf. Spending time with them made us notice how much of everyday life assumes you can hear and speak, like ordering at a counter, checking in at a clinic, asking a quick question. Things we do without thinking can take real effort for them, and often depend on someone else being patient or knowing sign language.

It made us realize how much easier we have it than a lot of people. We wanted to build something that could actually help people and have an impact.

What it does

Said With Hands is a sign recognition communication aid that runs in the browser.

Recognizes ASL fingerspelling and a starter set of signs from your webcam.
Learns your own signs in about 10 seconds. Perform a sign a few times and it's recognized right away, with no retraining.
Gemini turns uncertain recognitions into a clear sentence, labelled as a best guess with the raw signs always visible.
ElevenLabs speaks the sentence aloud.
Two-way: the hearing person's reply appears as live captions.
Private: no video ever leaves the device.

However, it is not yet is not a full ASL translator. ASL has its own grammar and relies on facial expressions.

How we built it

Next.js on Vercel. MediaPipe tracks 21 hand points per frame in the browser. Our matcher compares them against known letters and taught signs, using dynamic time warping for signs with movement. Gemini gets the top few guesses for each position and picks the sentence that makes sense. ElevenLabs handles speech, and the Web Speech API handles reply captions.

Challenges we ran into

Similar-looking letters (like M and N), keeping recognition stable without flicker, telling a real sign apart from random movement, and getting good results for people whose hands weren't in our training data.

What we learned

Recognition is never certain, so the useful question is what to do with uncertainty. Passing ranked guesses to Gemini instead of one answer made a big difference. We also learned to be honest about scope.

What's next

Testing with Deaf users, including our friend, and building it around their feedback. Also more signs, better handling of motion, and offline support.