Real Time American Sign Language (ASL) Translator

• 

  Home Page

• 

  Translator predicting correctly the gesture "O"

Inspiration

I wanted to create something that could possibly have a positive impact on some people. I think this project can do that, especially if I improve upon it in the future.

What it does

Currently, it takes webcam live input and outputs the ASL alphabet letter that is being signed by the users hand.

How we built it

Initially I had trained my own ML model on my machine locally and completely built the project, however I experiences some issues with my computer and my entire project was deleted. So due to lack of time I used a pre trained ASL image classification model and reimplemented the flask backend as well as the react frontend to utilize this model. The webcam input is done in react. The ML model used was made Marxulia on HuggingFace: https://huggingface.co/Marxulia/asl_aplhabet_img_classifier_v3

Challenges we ran into

When I trained my own model, I ran into issues with overfitting so I retrained with techniques to minimize overfitting. Another issue I faced was using webcam feed in react as it is something I had never done before. A third issue I ran into was determining when to send the image from the webcam feed to the backend to get a prediction.

Accomplishments that we're proud of

I am proud of developing a complete prototype in time, though it is still not as accurate as I'd like.

What we learned

I learned transfer learning when training my own model since I used transfer learning on a CNN. Another thing I learned is how to use webcam feed in react and how to pass that to the backend. Finally, it was my fist time implementing a ML in flask, and though it was very simple, I still learned many new things.

What's next for Real Time American Sign Language (ASL) Translator

Next is to train a more robust an accurate model. Another thing to work on is minimizing background noise since that tends to have a negative effect on the predictions. Finally, I would like to eventually have it be able to transcribe entire sentences.

Built With

• css
• flask
• html
• javascript
• jsx
• machine-learning
• python
• react