asl_interpreter

• 

  Demo

• 

• 

• 

Live Demo: https://mimetic-design-255618.web.app/#/

Inspiration

The inspiration for this project came from being in my ASL (American Sign Language) class. I found that ASL doesn't have all the resources that many other languages do. ASL is a full language like any other and isn't just hard-coded English. There is no online translator for ASL, and because ASL is a visual language it poses a unique problem. Many deaf people struggle to communicate in their everyday lives be it talking with family, going to the store, doctors' appointments, school, etc. Our goal was to create tools that would better assist the deaf to allow them to communicate, particularly through the automated interpretation of ASL.

What it does

Our project takes video input from a user and is capable of detecting ASL fingerspelling signs and transcribing it for the user live. We also created a web front that would allow it to be used from a variety of devices such as computers or phones just as long as it has a webcam.

How I built it

One of the largest parts of the project was collecting the data needed to train our machine learning model to be able to detect the signs. I took over 2600 pictures of my hand doing the letters of the ASL alphabet and then categorized them. These images were fed into a TensorFlow Deep Neural Network (DNN) training model to train the ML model on our images. We were able to achieve 99.7% accuracy on our TensorFlow model! We then built a script and web interface that can take video input from a user and respond with the letters that the user is spelling out.

Challenges I ran into

Since this was a hackathon, we had to be aware that we were working with limited time, meaning that we couldn't pick machine learning models that would take days to train. We had to be strategic and focus on building a larger dataset that would allow us to reduce training time, without the cost to the accuracy of our final model. We also had to find ways to remove unnecessary noise from our training data to speed up ML predictions.

Accomplishments that I'm proud of

Our main goal with this project was to be able to take live input from a user and quickly respond with what the user is saying, and with high precision. We we're able to do that and can respond to user input extremely quickly and accurately, to allow the user to focus more on communicating, rather than just trying to get the technology to work.

What I learned

This was the first time either of us had used computer vision or machine learning. We learned about what's required to make an accurate ML model and how to feed it with useful data, as well as what models are more useful in certain situations.

What's next for asl_interpreter

This project has so much room to be improved and expanded. A long term goal for this project would be to create the functionality that would allow for hand tracking to be able to represent signs that have a movement component. End-user applications could be created for all sorts of purposes like, for example, doctors' offices. The ability of a doctor to be able to communicate with their patients is extremely important to delivering care, and something like automated translation services, that would remove the need to request an interpreter, could greatly improve the lives and care of deaf patients. Then it could ideally be developed so it could just be used on a mobile phone so a signer could just sign to their phone and have it speak out what they are saying, allowing it to be used for all different kinds of applications.

Built With

• computer-vision
• google-cloud
• keras
• machine-learning
• matplotlib
• numpy
• opencv
• python
• tensorflow
• vue