ASL Interpreter

Inspiration

Our inspiration came from the communication gap that exists between the deaf and hearing communities. Many people who rely on American Sign Language (ASL) often struggle to be understood by those who don’t know sign language. We wanted to create a tool that uses modern computer vision and deep learning to help bridge that gap empowering inclusivity through technology.

What it does

ASL Interpreter uses a camera feed to detect and interpret hand gestures representing the letters of the alphabet. The model identifies the letter being signed in real time and displays the corresponding character on the screen, allowing ASL users to communicate more easily with non-signers.

How we built it

We used MediaPipe and OpenCV to capture and process hand landmarks from a live camera feed. These landmark coordinates were then passed into a Multilayer Perceptron (MLP) built with PyTorch, which was trained to classify different ASL letters based on the hand’s position and orientation. The frontend interface connects with the live camera feed to visualize the model’s predictions in real time, creating a smooth user experience for both ASL and non-ASL users.

Challenges we ran into

One of our biggest challenges was finding sufficient and balanced data to train the model accurately. We also struggled with data preprocessing, particularly ensuring the hand landmark coordinates were normalized and consistent. Finally, integrating the camera window with the frontend proved tricky — synchronizing live feed capture with the prediction output required careful handling of asynchronous processes.

Accomplishments that we're proud of

We successfully built a working prototype that can recognize multiple ASL letters with solid accuracy. Seeing the model correctly predict gestures in real time felt like a huge milestone, especially given the complexity of real-world hand movements and lighting variations.

What we learned

We learned a lot about the end-to-end process of building a deep learning application from data collection and feature extraction to training, evaluation, and deployment. We also gained valuable experience collaborating across machine learning and frontend development.

What's next for ASL Interpreter

We plan to expand our dataset to include word-level and sentence-level recognition, and eventually support full ASL translation. We also want to add speech synthesis, so recognized signs could be spoken aloud for even smoother communication.