Signal

Your hands, out loud.

Comment

Inspiration

Deaf and hard of hearing people get limited by something that has nothing to do with them. Most hearing people just never learned sign language, so a totally normal moment, ordering food, asking a teacher a question, talking to a doctor's front desk, doesn't work unless an interpreter happens to be there. And interpreters cost money and have to be booked ahead of time, so for anything spontaneous there's basically nothing.

I couldn't build a full sign language translator in a weekend, that's a real research problem with grammar and facial expressions and all of it. But fingerspelling can spell any word there is. Twenty six letters get you anywhere. So I figured if I could get a browser to read hand shapes fast and accurately off a regular webcam, that's a real doorway, even if it's not the whole language yet.

What it does

You sign at your webcam and Signal reads your hand, up to two at once, and shows you a live skeleton over it so you can see what it's picking up. You can fingerspell the full alphabet or use one of fifteen built in word signs like hello, thank you, water, help. A letter only locks in once you hold it steady, there's a little ring that fills up, so you don't get random doubled letters from your hand shaking a bit.

Finished words show up as captions on screen and get read out loud, so whoever you're talking to doesn't need to know sign language at all. If there's a sign you use a lot that isn't built in, you can teach it one, show it the sign, name it, and it remembers it from then on.

Everything happens on your device. No account, no upload, nothing sent anywhere.

How we built it

It's all client side. Next.js and TypeScript, deployed on Vercel. For hand tracking I used Google's MediaPipe HandLandmarker, which runs right in the browser through WebAssembly and gives you 21 landmarks per hand in real time, no server needed.

For classification I wrote my own k nearest neighbors model in TypeScript that works off those landmark coordinates instead of raw pixels. I liked that I could actually explain every part of how it makes a decision instead of it being a black box. Letters and word signs get classified separately so a fist shaped letter and a fist shaped word sign don't get mixed up.

Text to speech is just the browser's built in Web Speech API, no API key needed. Styling is all hand written CSS. Anything you teach it or calibrate gets saved in localStorage on your own machine.

Challenges we ran into

The biggest one was accuracy. Early on, if you tilted your hand or it was a mirror image of the training photos, accuracy dropped to around 29 percent, basically unusable. It only worked if you held your hand exactly the way the training data did, which isn't how anyone actually signs.

The fix was augmenting the training data with rotated and mirrored versions of every example before training. That alone took accuracy up to around 93 percent on hands the model had never seen at that angle before. That was the point where it went from a cool demo that only worked for me to something that could actually work for someone else.

Getting the hold to lock timing right took a while too. Too fast and every little hand twitch became a wrong letter. Too slow and it just felt annoying to use. Took a lot of trial and error to land on something that felt natural.

Accomplishments that we're proud of

Getting it to actually work on hands the model wasn't trained on, not just my exact hand at my exact angle, is the thing I'm most proud of. That's the difference between something that works once for a demo and something that could actually be useful to someone.

Keeping everything on device mattered a lot too. This app is watching someone's hands, and I didn't want to have to explain where that video goes, because the answer is nowhere.

What we learned

Reducing a hand down to 21 landmark coordinates does most of the work, way more than picking a fancier model would have. The real effort was making those coordinates hold up across different angles and hand positions, not making the model itself more complicated.

I also learned that scope matters more than I expected. It would've been easy to make this sound like it does full sign language translation and let people find out later that it doesn't. Being upfront in the app about exactly what it can and can't do felt like the right call.

What's next for Signal

I want to add more word signs past the fifteen it has now, and start handling two handed signs and simple motion signs instead of just static ones. Getting it to run well on a phone matters too, since most of these moments happen out somewhere, not sitting at a laptop. Longer term I'd like to get past just ASL, since this problem isn't specific to one sign language, it's specific to not having any doorway through it at all.

Built With

Share this project:

Updates

Submission history