SADO | Devpost

Inspiration

Uzbek Sign Language (USL) has very little digital support compared to spoken Uzbek — no real-time tools that let Deaf and hard-of-hearing people communicate with hearing people who don't know sign language, in either direction. Most existing sign-language tech is built for ASL or other major languages; USL speakers are largely left out. SADO (short for "imo-ishora tili" — "sign-gesture language") started from wanting to close that gap with something that actually works in real time, on a normal webcam, with no special hardware.

What it does

SADO is a bidirectional Uzbek Sign Language translator. It works two ways:

Sign → Text/Speech: point your webcam at your hand, sign a letter, and SADO reads the hand shape, converts it to text, and speaks it out loud. Text/Speech → Sign: type or speak a word, and SADO plays back an animated hand skeleton signing it, generated from real recorded motion data — not a pre-made video clip.

Everything runs with the camera feed staying on-device — only extracted hand-landmark coordinates (not video) are sent anywhere for prediction.

How I built it

The recognition side uses MediaPipe to extract 21 hand landmarks per frame from webcam video, feeding a RandomForestClassifier trained on landmark data extracted from real USL instructional video footage. The frontend is React (Vite + Tailwind), talking to a FastAPI backend that serves the trained model; MediaPipe itself runs client-side in the browser, so raw video never leaves the user's device. For the reverse direction, I record full landmark sequences (not single frames) for each word, and play them back as an animated skeleton on an HTML canvas. Speech input/output uses the browser's native SpeechRecognition/SpeechSynthesis APIs rather than a separate paid service. I built this solo, using Claude Code to accelerate implementation while I drove the architecture and debugging decisions myself.

Challenges I ran into

The biggest challenge was data leakage: an early version of the model hit 100% test accuracy, which turned out to be a red flag, not a win — a random frame-level train/test split let near-duplicate consecutive video frames leak across both sides, so the model was memorizing frames instead of learning the actual signs. It failed almost immediately on live webcam testing. Fixing this required a session-aware split (holding out whole video segments, not random frames) and wrist-relative landmark normalization to make the model robust to different hand positions and camera distances — which took real accuracy from a fake 100% down to an honest ~87%. I also had to catch a subtler bug: an illustrated hand graphic overlaid in the training video was at risk of being picked up by MediaPipe as a second real hand, contaminating the training data.

Accomplishments that I'm proud of

Catching the data leakage before it became a broken public demo, rather than after. Getting a real, honestly-measured accuracy number instead of a misleading one. Building a genuinely bidirectional translator — most student sign-language projects only go one direction — entirely solo.

What I learned

That a suspiciously perfect accuracy number is a bug report, not good news. That normalization and correct train/test splitting matter more to real-world model performance than almost anything else. And that reviewing AI-generated code carefully (I use Claude Code heavily) still matters — it introduced subtle bugs like a mid-recording label swap and an inefficient model choice (Holistic instead of Hands-only) that needed catching by hand.

What's next for SADO

Expanding past the alphabet into full word/phrase recognition, recording more signers/sessions for better generalization beyond one person's signing style, and eventually a fuller animated avatar for the sign-generation side.