Real-Time ASL Communication System

Inspiration

We were inspired by the everyday communication barriers faced by Deaf individuals who use ASL. Since most people do not understand ASL, we wanted to use AI technology could help make communication more accessible without relying on interpreters.

What it does

The system translates American Sign Language letters into spoken language in real time using only a standard webcam.

How we built it

We built a real-time ASL to speech system using a standard webcam. Using MediaPipe Hands, we tracked hand landmarks and trained the system on our own ASL alphabet dataset to recognize letters and convert them into spoken language.

Challenges we ran into

One major challenge was connecting the front end and back end of our system. While the front end correctly displayed the letters based on hand movements, it was unable to send this data to the back end. As a result, Gradium’s text-to-speech system could not receive any text, which prevented the speech output from working. We also faced issues with accuracy due to lighting, hand angles, and small differences in finger positioning.

Accomplishments that we're proud of

We successfully created a working real-time ASL to speech system without using specialized hardware and trained our own dataset from scratch.

What we learned

We learned how computer models track hand movement, how training data impacts accuracy, and how important data flow is between the front end, back end, for the external services to work.