Sign-Whisper

Sign Language Recognition in Videos for communication with and for the Deaf and Dumb

Comment

Inspiration

The mute living among us often find it hard to communicate with other people. We too fail to understand them most often due to lack of awareness about their mode of communication. Sign-Whisper was developed as a social good application to interpret the sign language to understand and better communicate with them.

What it does

Sign-Whisper detects the gestures used in the ASL (American Sign Language) and provides output through text and speech. A Machine Learning model identifies a gesture and outputs a word that it represents, to make us understand them, for example, the word 'Friend'. We increase the communication spectrum with the usage of GCP Text-To-Speech API to convert the text to speech, which enables who are hard of vision and reading to communicate with them, and the GCP Translate API helps in communicating in different languages too. A Web application does real time video capture of the gestures, and these are sent to the model to predict the word in real time. The outputs of the GCP APIs are also integrated into the WebApp.

How we built it

The Web Application, developed using HTML5 and CSS3 and Javascript integrates the real time video capture and the Machine Learning model. The model is built using CNN and RNN and is written in Python and Keras, and also incorporates Transfer Learning(cool feature right!). This model is trained using examples from the user and word predictions are displayed. The predicted word is given to the GCP Text-To-Speech API to convert it to speech. The text is also given to the GCP Translate API to enable communication with other languages. It will also output the speech output and the translated text output from the GCP APIs.

Challenges we ran into

  1. Finding appropriate dataset, collecting the ASL Dataset and preprocessing it.
  2. Integrating the GCP APIs to the pipeline of our model.

Accomplishments that we're proud of

Built the model

What we learned

  1. Working of GCP Text-to-Speech and Translate API
  2. Nuances of data pre-processing
  3. Combination of RNN and CNN for training the model to learn Sign Language
  4. Developing web-application as a UI Interface

What's next for Sign-Whisper

  1. Implementing spatial transformer network for improved gesture detection
  2. Enhancing the pre-processing steps further by better techniques

Built With

Share this project:

Updates