SignScribe

Inspiration

The inspiration for SignScribe came from my cousin, who is deaf. Growing up, I saw firsthand the communication barriers that exist between the deaf and hearing communities. While many people know a few signs, very few are fluent in American Sign Language (ASL), which often leaves deaf individuals at a disadvantage when it comes to day-to-day interactions. I wanted to build something that could help bridge this gap, creating a tool that not only empowers deaf individuals to communicate more easily but also makes ASL more accessible to people who don’t know it. The idea of real-time ASL translation felt both impactful and challenging—exactly the type of problem I wanted to tackle.

What it does

SignScribe is a system that translates ASL into text. Using a camera feed, it captures the hand gestures of a person signing and processes those gestures with a machine learning model. The model then predicts the corresponding word or letter and displays it as text on the screen in real time. In this way, SignScribe acts as a “bridge,” turning sign language into written words that anyone can understand, even if they don’t know ASL. The long-term vision is to extend this further into voice output or even reverse translation—where text or speech could be converted back into ASL avatars.

How we built it

We started by collecting and curating a dataset of ASL gestures. Each gesture needed multiple angles and lighting conditions to ensure the model could generalize well. Using Python and TensorFlow/Keras, we designed and trained a convolutional neural network (CNN) capable of recognizing different signs from the input frames. OpenCV was used to capture and preprocess the video feed, ensuring that hand shapes and movements were isolated before being passed into the model. We then integrated the predictions into a simple but functional GUI that outputs text in real time. The project required stitching together several tools: computer vision for hand tracking, deep learning for classification, and some frontend code to make the translation visible to users.

Challenges we ran into

One of the hardest parts was data collection and training. Machine learning does not "want to learn" unless the dataset is large, clean, and balanced. Getting accurate predictions required hours of fine-tuning, tweaking hyperparameters, and retraining. Another challenge was the natural variability in how people sign—even the same person may not make identical gestures each time. Making the model robust enough to handle that was a steep learning curve. We also had to optimize for speed; real-time translation demands low latency, so we worked hard to keep the pipeline efficient.

Accomplishments that we're proud of

We’re incredibly proud that we got a working prototype up and running. Watching the system recognize hand gestures and turn them into words was a huge milestone—it felt like the hours of debugging and trial-and-error paid off. We’re also proud of the dedication it took; we stayed up until 5 AM several nights in a row, pushing through roadblocks and helping each other stay motivated. Seeing our idea come to life, even in its first iteration, was deeply rewarding.

What we learned

This project taught us so much about machine learning, computer vision, and the patience required to get them working together. We learned how important dataset quality is, how to structure and preprocess data for a neural network, and how to balance model complexity with real-time performance. Beyond the technical skills, we also learned about teamwork, resilience, and how to break down a big, ambitious problem into smaller solvable steps.

What’s next for SignScribe

The future for SignScribe is endless. Right now, it can translate a limited set of ASL gestures into text, but we want to expand it to cover full sentences and more nuanced signing. We also see potential in adding reverse translation: converting written or spoken English into animated ASL avatars for a fully two-way system. Another exciting direction is deploying this as a mobile app, so people can carry it anywhere. With more training data and community feedback, we could make SignScribe an accessible tool that genuinely bridges the communication gap for millions of people worldwide.