Handterpret

Handterpret

Inspiration

With more than 400,000,000 hearing-impaired individuals worldwide, it is evident that hearing loss is a major issue in society. To address the severity of this issue, we created a translator from American Sign Language to English to help improve communication between individuals who cannot understand American Sign Language but speak English and individuals who are hearing-impaired using machine learning.

What it does

The translator opens a camera application that prompts the user to input hand signs and then translates the sign language to English text. Furthermore, the user may also convert the translated text to speech. The application does not require any specific hardware, just a computer with a camera!

The probabilities of the sign matching an intended character were calculated using a Naive Bayes algorithm. The Naive Bayes classifier obtained from this algorithm is combined with a decision rule, which maps the observation to an appropriate action.

How we built it

We used python to develop the frontend of the application. For the backend, we also used python in addition to opencv and the TensorFlow library to create the backend of the application. To help our convolutional neural network learn, we used the Keras library to train it by inputting images ourselves.

Challenges we ran into

This was our first time using the TensorFlow and Keras libraries, so it made it difficult to separate the image the camera is recognizing from the actual hand sign. Also, it was challenging to train the neural network because we had to input images that we created ourselves, which is extremely time-consuming. Moreover, hand signs that are similar, such as hand signs for the letter A and the letter E, are hard to distinguish and require more training.

Accomplishments that we're proud of

We are proud of creating an ASL translator application that does not use any specific hardware. Also, we managed to create a dataset large enough to recognize a lot of letters. Time was the biggest restraint, but we did what we could.

What we learned

We learned how to use the TensorFlow and Keras libraries in python and how convolutional neural networks work.

What's next for Handterpret

The obvious next step for ASL Translator is to improve the accuracy of our translator. Since our application is not restricted to specific hardware such as LeapMotion it works with any computer with a camera, so we hope to expand our application to mobile phones.