SignLingo

Inspiration

The idea started with motion-control video games. Seeing how cameras and sensors could understand human movement made us wonder: could this same technology be used for something genuinely useful?

That question naturally led to sign language.

Despite how powerful and expressive ASL is, there are surprisingly few public, non-developer tools that help people actually learn it. Most resources are passive, hard to follow, or inaccessible to beginners. Apps like Duolingo showed how language learning can be made engaging and accessible to everyone, so we asked: why doesn’t sign language have the same experience?

This project was born from that gap. By combining computer vision with an interactive learning flow, the goal is to help curious learners get started with sign language, and ultimately help bridge the communication gap between the deaf and hearing communities.

What it does

Our project is a Duolingo-style web app for learning sign language that uses computer vision to provide real-time feedback on user performance. Learners complete short, interactive lessons directly in their browser, where they are prompted to perform specific signs in front of their camera. Using computer vision, the system analyzes hand shapes, movements, and positioning, then compares them to the expected sign.

As users progress through lessons, the app keeps learning engaging through achievements, daily goals, and streaks that reward consistency and improvement. These features encourage regular practice while giving users a clear sense of progress over time. Combined with instant visual feedback, the app supports an active, learn-by-doing approach that helps users build confidence and accuracy as they continue learning.

By combining gamified learning mechanics with real-time visual feedback in an accessible web platform, the app makes sign language learning more interactive, motivating, and approachable, especially for beginners and self-learners who may not have access to formal instruction.

How we built it

We divided our team into two focused sub-teams. Kamilla developed the sign language recognition system using MediaPipe and scikit-learn. A dataset of 300 snapshots per letter of the English alphabet was collected and processed into hand landmark data using MediaPipe. This data was used to train a Random Forest classifier, achieving an accuracy of 99.6%. The model was first integrated into a local testing app using live webcam input to provide real-time predictions and feedback.

Meanwhile, Trista, Hung and Wen-Tzu built the frontend web app using React, creating an interactive and responsive interface. They implemented Firebase Authentication for user accounts and progress tracking. While the team attempted to use Firestore to store user progress and streaks, cloud storage could not be fully integrated during the hackathon, so data is currently stored locally.

To ensure reliability, a backup local interface was prepared in case frontend integration failed. Finally, the frontend and backend were connected using** sockets** and a locally hosted server for the sign recognition model, enabling real-time communication between the web app and the computer vision system.

Challenges we ran into

One core technical challenge was gesture recognition. Our ML model relied on distances between hand landmarks rather than hand orientation, which reduced accuracy for visually similar letters like K and L, exposing limitations in our feature engineering.

We also encountered frontend–backend integration issues with Firebase. Account deletions and forced refreshes frequently failed, and cloud storage of user progress was unreliable, so we temporarily persisted data locally. Implementing WebSockets for real-time communication required extensive debugging to achieve partial functionality.

Collaboration presented additional technical hurdles. Most team members were new to Git, GitHub, and full-stack development. We adopted a collective learning approach rather than isolated roles, leading to early merge conflicts and coordination challenges, but ultimately improving our understanding of the end-to-end stack. By hackathon’s end, we delivered a functioning product while strengthening our skills in debugging, integration, and distributed version control.

What we learned

Through this project, we gained hands-on experience with Git and GitHub by resolving multiple merge conflicts, and developed a solid foundation in machine learning with MediaPipe. We also strengthened our skills in frontend development, Firebase integration, and WebSockets. Extensive testing and iterative debugging not only improved our technical proficiency but also gave us practical familiarity with sign language gestures.