ASL Detector

Inspiration

Inspired by one of my classmates who use ASL to communicate, I wanted to create an application that would help me and others easily test themselves with the hand signs for easier communication.

What it does

The app quizzes the user on an ASL letter to show and if the user shows the letter in time, they will get it correct. The quiz ends once the user has successfully shown each letter.

How we built it

I used a training set of 80000 images from Kaggle hub that contained letters A-Z and some other miscellaneous signs. Then, I extracted each of the hand positions from each image using a library called mediapipe. After getting the proper data, I used that to train a neural network to recognize letters based on said hand positions. Finally after successfully training my model, I used the webcam to detect the user's hand position and predicted the letter with the model.

Challenges we ran into

At first, I tried to train the model only using images and not the hand coordinate points, but it was extremely inaccurate and took way too long to train. So, I switched to only taking the hand coordinates which gave me much better results.

Accomplishments that we're proud of

I was able to train a model to accurately predict most letters with minimal training.

What we learned

I learned how to track different poses using mediapipe, how to work with the OpenCV library, using numpy arrays, and training neural networks.

What's next for ASL Detector

I will use more data in training my model to make it more accurate. I'll also add in more signs and add in a feature to track letters that use motion as my project fails to capture that. I will also improve the user interface of the application as it's currently lackluster.

Built With

• opencv
• python