Sign-To-Text

The project is a proof of concept for a real time sign language interpreter. The project aim's to interpret the sign language captured through a camera/video.

Comment 2

Inspiration

Sign Language is what majority of the people who are part of the deaf and mute community use as a part of their daily conversing. Not everyone knows sign language and hence this necessitates the need for a tool to help others understand Sign. This basically emphasizes the possibility for the deaf and mute to be solely independent and not in need of translators when they have to address or even have a normal conversation with those who do not understand sign language.

What it does

The platform basically takes in input via a camera of a hand gesture and tells you which alphabet does the letter stand for. It is based on the American Sign Language conventions and can recognize all the alphabets given the conditions are met which are derived from it's training data.

How we built it

The backend was done using Python-Flask, Tailwind CSS was used for frontend development along with HTML and JS. For the AI part , Microsoft Azure Custom Vision service was used. The custom vision service can be used to train and deploy models with high availability and efficiency.

I used the ASL image dataset from Kaggle where 190 random images from the entire ASL was taken and used to train the model for each alphabet. Hence a total of 190*26 images were used to train the Azure Custom Vision Model.

The application has been separately deployed on Azure WebApp service with GitHub Actions auto redeploying on new commit using a simple CI workflow.

Challenges we ran into

  1. Securing the critical keys in the code before pushing to github
  2. Bottleneck on model efficiency when it comes to the use of real time data.
  3. Azure Limitations on 5000 images per Custom Vision Project.

Accomplishments that we're proud of

  1. Making a model that successfully classified the ASL test data from Kaggle.
  2. More deep understanding of Azure technologies and cloud.

What we learned

  1. Frontend Development with Tailwind CSS
  2. Integrating Azure Services into Python Flask
  3. Deployment on Azure

What's next for Sign-To-Text

  1. More efficient model.
  2. Real-Time sign to text conversion followed by a text-to-voice converter.
  3. Sign-To-Voice Converter

Built With

Share this project:

Updates