AirSign

Inspiration

Sign language is so much more expressive than typing. Your face, your hands, how fast or big you sign, all of it carries meaning and emotion. But when Deaf or non-speaking people need to talk to someone who doesn't sign, they usually have to fall back on typing on a phone or writing things down, and a lot of that gets lost. Hearing people get to say things with tone and emotion without thinking about it, and we wanted signers to have that too.

So we set out to build a bridge between people who sign and people who don't. AirSign recognizes your signs and says them out loud, and when the other person talks back, you see what they said. It's not a perfect interpreter, but it's enough to get a conversation going in both directions.

We also wanted it to help people actually learn. That could be a young Deaf kid learning to sign for the first time, or a parent, family member or friend learning so they can communicate with someone they care about (or learning right alongside their kid). Practicing alone is hard when there's nobody to tell you if you're doing it right, so AirSign doubles as a practice tool that gives you that feedback.

What it does

AirSign watches your hands through your webcam and speaks what you sign out loud. Hold a sign until the ring fills up, it locks into a sentence strip, and it gets read out in a natural voice. When the other person replies, AirSign shows live captions of what they said.

It can recognize:

ASL words: around 250 of them
Fingerspelling and numbers: A–Z and 0–10
Quick phrases: simple gestures for everyday things like "Please wait", "Thank you" and "I need help"
Your own custom signs: record a gesture, give it a meaning, and AirSign learns it on the spot

The Practice tab gives you random letters and numbers to sign, tells you what the camera thinks you're doing, and lights up green when you get it right.

Everything runs in your browser. Your video never leaves your device.

How we built it

Next.js, React, TypeScript and Tailwind for the app
MediaPipe for hand tracking, plus its Holistic model (hands, face and body) for ASL words, since a lot of signs depend on more than just your hands
ONNX Runtime Web to run the ASL word model and a pretrained alphabet model right in the browser
A kNN classifier on our own labelled dataset for the alphabet, and geometry rules for numbers
ElevenLabs for the voice and the Web Speech API for live captions
Deployed on Vercel

Challenges we ran into

Getting a first version working was honestly only half the journey. The other half was fine-tuning everything until it actually felt usable.

ASL words were way too rigid at first. We were waiting for one really confident guess, which almost never happened, so real signs just got ignored. We changed it so a word counts if it keeps showing up near the top across a few guesses in a row, even at lower confidence. That made a huge difference.
Flicker. Raw predictions jump around a lot frame to frame. We added a hold-to-confirm ring, cooldowns and an adjustable repeat delay so AirSign only speaks when you actually mean it.
Getting the word model's input right. It expects 543 points per frame in a very specific layout, and our left and right hands kept getting swapped because the camera preview is mirrored but the actual frames aren't.
Performance. Running these models in a browser at a decent frame rate is tough, so we only run one heavy model at a time and added a GPU/CPU toggle for laptops where the GPU gives junk results.

Accomplishments that we're proud of

Custom signs. You can record your own gestures, give them any meaning, and build your own vocabulary. Every family or friend group has their own signs, and AirSign can learn them in a few seconds.
Real two-way conversation: signs become speech, and speech becomes captions
Five recognition modes running live in the browser, with no video sent anywhere
It works on phones, including the rear camera, so you can point it at someone else

What we learned

We learned how much of a model's accuracy comes down to tuning: how you prepare the input, how long you wait, how confident you need to be. A slightly less accurate model that's tuned well felt way better than a rigid one. We also learned that making something feel reliable matters just as much as raw accuracy.

What's next for AirSign

More sign languages. There are hundreds of sign languages around the world, and ASL is just one of them. Adding BSL and other widely used sign languages is a big next step for us.
Better learning and practice features, like embedded videos showing how to sign each word or phrase
Better classifier models so AirSign can interpret and practice a lot more words
Turning sequences of signs into full, natural sentences