Signboard

Interface
Explanation
First page
More context

A friend of ours grew up with deaf parents, one partially deaf and one fully deaf. Because she could always speak with the parent who could hear some sound, she never learned ASL. When she finally set out to learn it, she found it far harder than she expected. ASL isn't English on the hands. It's a visual, spatial language with its own grammar, and some of its most essential features, like classifiers, have no English equivalent at all. She's improving because she has a teacher who watches her hands and corrects her in real time. Most learners don't have that. They have videos, which can show a sign but can't tell you when you're getting it wrong. That gap became SignBoard. It teaches ASL classifiers, the handshapes signers use to represent objects and act out scenes in space. Using your webcam, it recognizes three core classifiers: a vehicle, a person, and a flat surface. As you move your hands, a comic-book-style scene comes to life on screen, and you work through short challenges that build on each other, from making a car, to driving it uphill, to crashing it into a wall. If your handshape is right, the scene responds instantly. If it isn't, nothing moves, so you always know where you stand. To make that feedback possible, we built a browser-based app on MediaPipe's Hand Landmarker, which tracks 21 3D points on each hand in real time. Rather than training a model, we wrote geometric rules for each classifier using MediaPipe's 3D world landmarks, which stay reliable even when fingers point toward the camera. We check which fingers are extended, calculate palm direction from points on the wrist and knuckles, and designed the math to work for either hand. The flat-surface classifier also checks orientation, so a wall requires fingers pointing up, just as it would in ASL. A short persistence window keeps handshapes from flickering, and the comic-print visuals, with halftone textures and bold ink outlines, reflect the way signers stage scenes like a storyboard. Getting there wasn't simple. ASL is a language of movement, and the same handshape can mean different things depending on whether it's held still or traveling through space, so we had to track not just what each hand looked like but how it moved. Every person's hands are also different in size, flexibility, and how they naturally form a shape. We had to make recognition forgiving enough to work for anyone while still precise enough to tell handshapes apart, and the thumb, the least stable point to track, turned out to be what separates a vehicle from everything else. What we're proudest of is how approachable the result feels. Learning ASL from videos can be overwhelming because everything comes at you at once. Signboard breaks learning into small, sequential steps, each building on the last, so progress compounds without ever feeling insurmountable. The interface is simple enough to understand at a glance and interactive enough that learning feels like play. Along the way, we learned that building for a language means respecting how it actually works. We started out thinking of ASL as a set of gestures to detect, but classifiers showed us that meaning lives in handshape, orientation, and movement all at once. We had to research details like which hand moves and which way a wall faces, because an app that teaches a language slightly wrong does more harm than good. Next, we want to expand to the wider set of classifiers signers use every day, add two-person scenes so learners can tell stories together, and have learners sign the noun first, like CAR, before the classifier takes over, just as fluent signers do. Most importantly, we want to build the next version alongside Deaf signers and ASL teachers. Our friend learned because someone was watching her hands. We want everyone who wants to learn ASL to have that, and to finish their first lesson wanting to keep signing.