Sign2MeetU

Training the sign for want
Training the sign for where
Training the sign for wifi
Logo

Inspiration

More than 70 million Deaf people worldwide use sign language, but most hearing people don’t know it, and an interpreter isn’t there for everyday moments like ordering a coffee, finding a lecture hall, or telling a nurse where it hurts. One of our members who conducts research UW's AHSLAB brought up the lack of easily accessible translation tools for ASL (American Sign Language). Existing sign-recognition demos treat signing as a word-by-word code. But ASL depends on context: signers point at things around them, and a sign like “THAT” only makes sense if you can see what they’re pointing at. We wanted a translator that understands both the sign and the environment.

What it does

Sign: the signer signs into the phone camera. Our own classifier, trained on a dataset our team recorded, recognizes each sign from hand movements tracked in the browser. Understand the scene: Perceptron Mk1.5 looks at the signer’s surroundings: what kind of place it is, the objects nearby, readable signs, and what the signer is pointing at. Speak: Perceptron turns the recognized signs and the scene into one natural English sentence, which is shown on screen and spoken aloud.

How we built it

Frontend: Next.js, React and Tailwind, deployed on Vercel as a mobile web app. Hand tracking: MediaPipe Hand Landmarker runs in the browser and finds 21 points on each hand in every video frame. Our classifier: we built a /train page to record labeled examples, and recorded [N] examples across [3] signers. Each recording becomes a “fingerprint”: hand shapes measured relative to the wrist, plus hand location, resampled to 16 frames. A k-nearest-neighbors classifier with dynamic time warping compares fingerprints even when people sign at different speeds. Perceptron Mk1.5: used through Perceptron’s API with structured JSON output. One call describes the scene (place, objects, readable text, pointing target). A second call turns the sign sequence plus the scene into fluent English, without adding meaning that wasn’t signed. Speech: the browser’s built-in text-to-speech.

Challenges we ran into

Sign language isn't only in the hands. In ASL, facial expressions carry grammar. Raised eyebrows can turn a statement into a yes/no question, and expressions change a sign’s meaning or intensity. Our classifier tracks hand landmarks only, so we struggled to capture these nuances reliably. Camera angle, lighting, background, and skin tone can all affect how well hands are detected and signs are recognized. A sign that’s recognized perfectly at one angle can fail at another. We normalized hand positions so size and distance from the camera matter less, but viewing angle and lighting was a big consideration in implementation.

Accomplishments that we're proud of

We’re proud to have built an end-to-end prototype that brings together camera capture, sign recognition, confidence feedback, speech output, and a growing transcript. We also made the experience work with a phone camera, without requiring dedicated hardware.

What we learned

Language is full of nuance. We started out thinking of translation as mapping signs to words. We learned that meaning also lives in context, pointing, word order, and facial expression, and that a good translator has to understand all of it. Trade-offs between time and compute. Our classifier runs on the phone in milliseconds at no cost, but only knows what we taught it. Perceptron understands far more, but each call goes over the network and costs money. Deciding which part of the problem each one should handle taught us to think about speed, cost and accuracy together. In the end we learned to design around those factors: our classifier handles a focused vocabulary of 13 signs, and Perceptron fills in the rest from context.

What's next for Sign2MeetU

We plan to expand the vocabulary and improve recognition across different styles and environments. Fingerspelling for names and places. Two-way conversation: speech-to-text so the hearing person can reply. Building it with the Deaf community, with their input in mind.