SignMate

Most ASL apps are just video dictionaries. We built a real time sign language tutor that gives you instant feedback. Our goal is to make learning sign language accessible, confident, and inclusive.

Comment

Inspiration

Learning sign language is difficult without feedback. Most tools rely on videos, which do not tell you if your hand shape or movement is correct. We wanted to build something that could act as a real time guide, helping learners practice with confidence and communicate more effectively.

What it does

We built a real-time sign-language interpreter that detects what letter (A-Z) or number (0-9) you are signing. The system displays both the predicted sign and a confidence score so users know how accurate their signing is. Additionally it can detect gestures and words such as “hello” and “me”. Users can opt to upload a signed video and have it instantly translated into text, making communication faster and accessible.

How we built it

  • We built our MVP using Python as our core engine. We integrated Mediapipe for hand landmark tracking, allowing us to capture 21 distinct 3D points on the hand without specialized hardware. For the 'brain' of our tutor, we implemented a custom PyTorch neural network to identify letters and numbers based on landmark coordinates.

  • Trained a custom PyTorch model to classify letters and numbers, and reused the same model for both real-time camera input and uploaded video analysis.

  • Developed a Flask backend to handle live video streaming, buffering, inference, and offline video transcription through a shared ML pipeline.

  • Created a React frontend that displays live predictions, confidence scores, fingertip motion graphs, and translated text in an accessible, user-friendly interface.

  • Learned and implemented the full stack including data collection, model training, and deployment during the hackathon.

Challenges we ran into

When creating the AI model, we ran into several issues. The first and most substantial was figuring out how to get large amounts of data to train our model. Without enough, it won’t be able to detect letters and numbers in a wide range of scenarios. We found an open source dataset of 223,000 images of letters A - Z. We then wrote a script to convert each image into a proper data numpy array for model training.

Secondly, we realized that our model was overfitting the data. This means it was tuning its weights and biases too close to the training data, so much so that when it was given something completely new, it gave random / garbage values. To counteract this, we introduced a dropout rate, where a certain percentage of random data arrays would not be used for a specific epoch. We also tuned parameters such as the epochs and the number of neurons in each hidden layer. This eventually allowed us to get a model that trains well with the given data without overfitting, and is able to detect letters and numbers accurately in a wide range of situations.

Accomplishments that we're proud of

We are proud that our final product has a clear real-world use case: helping people practice sign language with immediate, objective feedback rather than passive video learning. By turning a webcam into an interactive tutor that evaluates hand shapes and movements in real time, SignMate lowers the barrier to learning and makes practice more effective and accessible. We’re also proud that we built a complete end-to-end system within the hackathon timeframe by combining our strengths in coding, machine learning, and design.

What we learned

This hackathon provided us with lots of experience in computer vision and pytorch AI model creation. We learned about how data is processed in openCV, and how crucial large amounts of varying data is for a model. We fell into maybe pitfalls and learned how to fix them every time. From making the mediapipe to work properly, to getting training data for the AI, and fixing the overfitting issue. We are much more knowledgeable in computer vision and AI concepts now than we were before.

What's next for SignMate

We will continue to develop several core features such as gestures for whole words, and forming sentences. We will also add in a tutor feature which actively teaches the user how to make better signs, based on how they are currently forming them.

Built With

Share this project:

Updates