Signly

Inspiration

When you want to learn French, Spanish, or another spoken language, it’s easy to find an app that helps you get started. We wanted that same approachable experience for someone learning American Sign Language. Maybe you have a friend with a hearing disability, family member, classmate, or coworker and want to communicate more directly. You might also just want to learn ASL. Finding videos is one thing, but knowing whether you’re doing a sign correctly is another. We built Signly to make those first steps less intimidating, with short lessons, games, and feedback while you practise.

What it does

Signly turns your webcam into a practice tool for ASL fingerspelling and numbers. You follow a handshape guide, try the sign, and get immediate feedback. Lessons, speed challenges, and math games give you different ways to practise, while XP, streaks, and a leaderboard help you keep going. We also built a searchable dictionary with 122 words and video examples from the ASL Citizen dataset. These examples introduce everyday vocabulary beyond the alphabet. The camera currently checks letters and numbers; it does not evaluate full-word signs or fluent ASL sentences.

How we built it

We built the frontend with React, TypeScript, Vite, and Tailwind CSS, splitting the work across recognition, gameplay, voice, and backend features. For recognition, we use MediaPipe Hand Landmarker to track 21 points on the hand. Our classifier examines finger extension, thumb position, contact, and spacing to identify static letters and numbers. A sign must remain stable for about a second before it counts, so a passing guess doesn’t immediately earn points. J and Z needed a different approach because they involve movement. We track the fingertip’s path over time and check its strokes. We also added a small TensorFlow.js motion model, trained on synthetic trajectories, with the geometric detector available as a fallback. Gemini provides short coaching messages from landmark summaries through a server-side API. ElevenLabs powers spoken prompts and voice answers in math games. Supabase supports profiles and leaderboard data, and we host the app on Vercel. Camera recognition runs in the browser, and coaching receives landmark summaries rather than raw webcam images.

Challenges we ran into

Recognition was our hardest problem. Several signs have similar handshapes, and a small change in thumb position can matter. Camera angle, lighting, and fingers blocking one another made those differences harder to measure. Movement brought another set of problems. Our first J/Z detectors passed tests using but struggled with actual webcam gestures. We had to account for uneven speed, pauses, tracking noise, and finger flexion during a sign. That taught us to treat synthetic tests as a starting point, rather than proof that recognition works for real people.

Accomplishments that we're proud of

We’re proud that we built a working loop where you can see a sign, try it on your webcam, and get feedback right away. Connecting recognition, coaching, voice, and scoring made Signly feel like a learning tool rather than a collection of separate features. We also went beyond alphabet practice, adding a 122-word video dictionary, guided lessons, speed challenges, and math games.

What we learned

We learned how much work sits between detecting a hand and understanding a sign. Tracking landmarks gives us coordinates; turning those coordinates into useful feedback requires careful rules, timing, testing, and clear limits. We also learned that teaching ASL goes beyond the alphabet. Facial expression, movement, body position, and space all carry meaning. Our current app is a beginner practice tool, and expanding it responsibly will require feedback from fluent ASL signers and educators.

What's next for Signly

We want to collect real gesture examples from a wider range of signers, improve recognition across camera setups, and work with educators to review the lessons. From there, we hope to add more movement-based signs and lessons that introduce ASL grammar in context.