ASL Active Learning

• 

  User Interface

• 

  Live Working of the algorithm

The video is kept silent for the deaf community and working with visuals!

Inspiration

In the United States approximately 600,000 people are deaf and approximately 6,000,000 people report having a lot of trouble hearing.
On a Duolingo (A CMU based startup) forum from 2019 it was said that “an odd number of requests were being sent for sign language to be incorporated.”. However, the biggest issue is: “how Duolingo could possibly view a person's signs to inform the user if his signing is correct or not.” We wanted to create a beta model (with limited functionality to the sign language alphabet) that could do just that.

What it does

Using your webcam, it detects the hand gesture of the user and puts forth if the user has done it correctly.

How we built it

We built the machine learning classification model using tensorflow to process the sign language images. The CNN (Convolutional Neural Network) was trained on over 27,000 images (dataset can be found on Kaggle.com). Real time image detection was done using a technique called YOLO (You Only Look Once) to create the hand frame.

Challenges we ran into

he biggest issue with the real time implementation was locating the hand so the image could be cropped to just show the hand (this was the style of the images in the kaggle dataset).
Typical OpenCV methods don’t work and R-CNN’s are slow. The solution was to use YOLO object detection.

Accomplishments that We are proud of

We created an entire model, which integrates openCV and higher computer vision techniques with easy to use interface in a matter of 2 days. The platform will be useful at a larger level with higher social impact.

What's next for Active Learning for Sign Language Using Computer Vision

Possible extensions of the model include: Rating the performance of the user based on a score Voice-added tutorials Mobile application for Android,Apple Social Media Interactions Saved user data in the application Leaderboard to keep track of your progress DuoLingo type of interface

Built With

• keras
• opencv
• pandas
• python
• tensorflow
• tkinter
• yolo