SignMeUp

Screen capture to practice letters
Word of the day challenge
Main page

Inspiration

Most people have never learned a single sign, not because they don't want to, but because existing resources are dry, inaccessible, and offer zero feedback. We wanted to change that. Inspired by the 70+ million deaf and hard-of-hearing people worldwide who use sign language as their primary form of communication, we asked ourselves: what if learning ASL felt less like studying and more like playing a game? SignMeUp was born from the belief that the barrier to sign language isn't ability, it's engagement.

What it does

SignMeUp is a real-time ASL learning web app with two modes. In Alphabet mode, users practice all 26 ASL letters one at a time, earning XP and building streaks as they go. In Word of the Day, a fresh word is generated daily by Gemini AI users, who finger-spell it live in front of their camera to claim a 50 XP bonus. The app tracks 21 hand landmarks in real time using MediaPipe, classifies the sign being made, and gives instant feedback. A reference panel shows the next letter, along with a Gemini-powered tip explaining exactly how to form it. When you nail a word, ElevenLabs reads it back in a natural human voice, making the feedback loop feel alive rather than robotic. Miss too many letters and the app tells you precisely what to fix and how.

How we built it

MediaPipe Hands for real-time hand landmark detection at 30fps directly in the browser
Custom ASL gesture classifier built on top of MediaPipe's 21 landmark coordinates to recognize all 26 letters
Gemini API for daily word generation, per-letter coaching tips, and end-of-session summaries
ElevenLabs TTS API for natural, human-sounding audio feedback on every completed word
React + Tailwind CSS for the UI, with a retro pixel-art game aesthetic
Lovable for rapid UI prototyping and component scaffolding
No backend required, the entire app runs in the browser ## Challenges we ran into ASL gesture classification is genuinely hard. Many letters look visually similar (H and U, A and S and E), and small differences in hand angle or lighting can throw off the classifier entirely. Getting reliable real-time detection without false positives required significant tuning of the landmark geometry logic. We also had to carefully merge two separate codebases, a Lovable-exported React UI and a vanilla JS logic layer, into one coherent app without breaking either side. Syncing the MediaPipe hand detection lifecycle with React's component state was another unexpected challenge that cost us several hours. ## Accomplishments that we're proud of We're proud that the app actually works in real time. You hold up your hand, the skeleton renders instantly, and the letter gets recognized within seconds. The integration between all three AI APIs, MediaPipe, Gemini, and ElevenLabs, running simultaneously in a single browser tab with no backend, is something we're genuinely proud of. We're also proud of the UI: it looks like a game you'd actually want to play, not a school exercise. And we built it all in under 24 hours. ## What we learned We learned that real-time computer vision in the browser is far more accessible than we expected. MediaPipe abstracts away most of the complexity. We also learned how powerful it can be to combine multiple AI APIs: each handles a narrow task (see, coach, speak), and together they create an experience that feels cohesive and intelligent. On the product side, we learned that scoping ruthlessly is the difference between a working demo and a half-finished idea. We cut a lot of features to ship something solid, and that was the right call. ## What's next for SignMeUp
Expanding beyond fingerspelling to full ASL words and common phrases
A persistent user profile with XP history, level progression, and badges
Multiplayer mode (sign-off challenges between two players in real time)
Mobile support so you can practice anywhere, not just at a laptop
More languages beyond ASL(American), BSL (British), LSF(french), and other regional sign languages
Partnering with schools and accessibility programs to bring SignMeUp into classrooms