SelfSign

Inspiration

We were familar with a project that tracked hand movements and immediately thought of creating SelfSign!We wanted to create a tool that allows users to actively practice ASL rather than passively memorize it, using real-time feedback to support learning. That way users can accurately learn without the help of a course or tutor.

What it does

SelfSign is an interactive ASL learning tool that uses the user’s webcam and MediaPipe hand tracking to recognize ASL alphabet gestures in real time. Users progress through the alphabet step-by-step, practicing three letters at a time. After each checkpoint, they can choose to review or take a quiz to reinforce their knowledge. Once all letters are completed, users take a final exam covering the entire alphabet.

How we built it

We built SelfSign using React and Vite for a fast and responsive frontend, Material UI for a clean and consistent user interface, and MediaPipe for real-time hand tracking and gesture recognition. We also implemented custom logic to distinguish between similar ASL hand shapes and to track motion for dynamic letters like J and Z.

Challenges we ran into

One of our biggest challenges was accurately distinguishing between similar hand shapes (such as A, E, S, and T), which required careful tuning of detection logic. Additionally, we were learning ASL ourselves while building the project, which made it both challenging and rewarding to translate real gestures into code.

Accomplishments that we're proud of

We are proud of successfully building a real-time ASL recognition system, creating a structured learning experience with checkpoints, quizzes, and a final exam.

What we learned

Through this project, we learned how to work with MediaPipe, how to handle gesture recognition challenges in a browser environment, and how to build a full interactive learning flow in React. (Funfact: We also now know the entire alphabet in ASL through building this!)

What's next for SelfSign

In the future, we plan to expand beyond the alphabet to include words and phrases, improve gesture recognition accuracy (for J and Z), add progress tracking and personalization, and continue enhancing accessibility features.