CelestiSign

Inspiration

More than 90% of deaf children are born to hearing parents, and most of those parents don't know sign language. For many deaf children, the first communication barrier isn't at school or work. It's at home.

It continues into adult life. Around 70 million deaf people worldwide use sign languages, but most hearing people around them can't sign. In the US, deaf adults are employed at a rate more than 20 points lower than hearing adults, even though full-time deaf workers earn about the same. The gap isn't ability. It's communication.

People want to learn: ASL is now the 3rd most-studied language at US colleges. But learning alone is hard. A video can show a handshape, but it can't tell you if your hand is right. Classes cost time and money, and practising alone gets boring.

So we asked: what if learning to talk with a deaf friend, classmate or child felt like playing a game together?

What it does

CelestiSign is a two-player space game where your hands are the controllers. Two players share one webcam. Asteroids carrying letters fall toward each player's base, and you sign the letter in ASL to blast them.

Versus mode: both players get the same asteroids, so the better speller wins. Power-ups like Shield, Freeze and Leech add strategy.
Co-op Boss mode: team up against THE SILENCER, an eye that drowns out communication. Its core only breaks when players take turns signing a chain of letters.
3D handshape coach: a ghost hand shows each letter with your live hand on top. Every finger turns green, amber or red, with feedback like "Adjust: thumb, index."
Hands-free menus: hold B to confirm and L to go back. You never need the keyboard.
A boss that talks back: Google Gemini can write the boss's taunts live, teasing players about letters they keep missing.
Private and accessible: recognition runs on your device, so no video ever leaves your computer. There's also a keyboard mode and a reduced-motion setting.

Play it: https://celestisign.64.177.46.240.sslip.io/

How we built it

MediaPipe tracks up to two hands from one webcam.
Our own TensorFlow/Keras model, trained on 3,864 samples from three public datasets, recognises 24 static ASL letters. It reads hand landmarks instead of images, so it's tiny (about 100 KB) and fast.
TensorFlow.js runs the model in the browser. A letter only counts after it's held steadily with enough confidence, which prevents accidental hits.
JavaScript, Vite and Canvas power the game, three.js the 3D coach, and GSAP the animations.
Web Workers run hand tracking in the background so gameplay stays smooth.

The model scores 96.4% in cross-validation and 86.6% on a dataset it never saw, so the game uses only the 14 most reliable letters.

Challenges we ran into

Left hands: most training photos showed right hands, so left-handed players were recognised worse. We flipped the photos to train on real left-hand data, and mirror left hands during play.
Real-world accuracy: 96.4% in testing dropped to 86.6% on new data. Instead of hiding it, we designed around it by using only the letters that stay reliable.
Two players, one camera: we assign hands by screen side, hold a player's spot if their hand briefly leaves the frame, and pause the game if it's gone too long.
Speed: hand tracking took ~90 ms per frame on a slow laptop. Moving it to a Web Worker cut the main-thread cost to ~0.3 ms.
Python vs JavaScript: the model is trained in Python but runs in JavaScript. We wrote tests to make sure both give the same predictions.

Accomplishments that we're proud of

We built our own recognition model end to end and reported its weaker numbers honestly.
Two people can play with one webcam and no controllers, even in the menus.
Our coach tells beginners exactly which finger to fix, not just "wrong."
Everything runs privately in the browser, with nothing to install.
70 automated tests, and the game is live for anyone to try.

Most of all, a project that started as "can we recognise hand signs?" became "can we help someone talk to a deaf person in their life?"

What we learned

ASL is a complete language, with its own grammar, facial expressions and culture. Fingerspelling is only a small part of it, mostly used for names. That's why we see CelestiSign as a first step, not a replacement for learning the language.

On the technical side, we learned that a high accuracy score isn't the same as a good experience. Speed, reliability and clear feedback mattered just as much as the model.

Above all, we stopped asking "What can we build with AI?" and started asking "Who is this for, and does it help them?"

What's next for CelestiSign

Beyond letters: add motion letters (J and Z), then whole signs, then full sentences.
Built with the Deaf community: work with Deaf signers and ASL teachers to make CelestiSign accurate and respectful.
Connect people: online play with friends anywhere, and a family mode for hearing parents of deaf children.
Sustainable, not greedy: free for individuals and families, always. Schools and ASL programs pay for classroom features like teacher dashboards and progress tracking, which funds development without charging the people who need it most.

One game can't remove every barrier, but it can make someone curious enough to learn the next sign.

We don't want technology to speak for people. We want it to help more people understand each other.