AI Sign Language Teacher

Inspiration

The idea for AI Sign Language was inspired by the need for accessible and interactive resources for people who want to learn sign language, whether for personal interest, professional development, or to communicate with loved ones who are Deaf or hard of hearing. Traditional learning resources often require in-person instruction or expensive programs, which aren’t feasible for everyone. We wanted to leverage AI and machine learning to make sign language learning more interactive, accessible, and fun for learners.

What it does

AI Sign Language provides an interactive platform where users can learn American Sign Language (ASL) through AI-driven video analysis, and real-time feedback. Users can learn simple sign positions and then practice by recording through their webcam. The AI system analyzes the user's gestures, compares them with correct signing techniques, and improve accuracy.

How we built it

The platform was developed using a combination of front-end and back-end using react typescript and python respectively, with a focus on machine learning and computer vision for gesture recognition. Using Python and TensorFlow a custom-trained model trained by our hackers can recognize ASL signs from a live video input. The model was trained on a dataset of frames of data points provided by google mediapipe which we refined into our required hand signs. For the website itself, we used React for a responsive and interactive user experience, and Flask for server-side handling of video uploads and real-time feedback processing.

Challenges we ran into

Building an accurate gesture recognition model for ASL presented several challenging problems. Firstly, the biggest problem we had was how to interpret and differentiate hand signs of similar looks. This can be solved by having our model take in more data set and tuning the wireframe generation through google mediapipes built in features. Additionally, it was a challenge for us to create a data pipeline from the model and to the website. But were able to successfully create a connection using rest api. However this also created a new problem of processing efficiency and heap-stack memory overload due to too many api requests being made. The problem was we could not fully process a pixel matrix worth of data at the speed of multiple requests per second. Our solution was to manually create a delay between each request which would slow down the processing but prevent a memory overload.

Accomplishments that we're proud of

We’re particularly proud of achieving a high level of accuracy in sign recognition, which was accurate enough to differentiate different gestures. Creating an interactive learning experience that feels responsive and personalized gave us satisfaction as was developing a platform that we hope will lower barriers for ASL learners. We're also proud of creating an inclusive tool that could help bridge communication gaps and promote a more accessible world.

What we learned

We gained valuable experience in machine learning, particularly in working with gesture recognition and video processing. This project taught us the importance of data diversity and model tuning when working with human movements and how essential it is to balance accuracy with real-time performance for a positive user experience. Additionally, we learned about the intricacies of ASL and the unique considerations in designing for accessibility, especially for Deaf and hard-of-hearing communities.

What's next for AI Sign Language

In the future, we hope to expand the platform to support additional sign languages from around the world, enabling people to learn and practice sign languages that are regionally specific. We also plan to enhance the AI’s capabilities to understand more complex phrases and sentence structures, enabling users to learn not only vocabulary but also grammar and conversational skills in ASL. Additionally, we’d love to incorporate a social feature, where users can practice signing with each other in real-time, and to partner with Deaf educators and organizations to continuously improve and expand the platform.