We decided to do something with visual recognition. After much deliberation, we decided to do a sign to text converter. We wanted to create something that would help people who don't know American Sign Language (ASL) to be able to converse with vocal and hearing impaired people.

What it does

The Raspberry Pi takes input of the hand gestures and return a letter the hand gesture corresponds to. A series of such hand gestures would form words and sentences. We decided not to do any word gestures, and we thought it would be impractical to do that in 36 hours.

How we built it

We used a Raspberry Pi and install the Raspbian OS on it. Using a Pi Camera Module, we were able to capture images on the Pi. We decided to use simple images and the Clarifai API for the hand gestures. We used multiple images sources (~10 for each alphabet) to train our model to identify hand gestures. Using a python program, we trained our model to make probabilistic guesses on the hand gestures. Doing this multiple times, it would string the letter together to form words and sentences.

Challenges we ran into

Initially, we decided to use a XBox Kinect but the hardware wasn't available. We then decided to use a Leap Motion, but all of them were taken. Finally, we decided to use images. We had a bit of problem installing the OS on the Raspberry Pi, and had to eventually get a new SD card and re install the OS. For presentation, we decided that we would use a website to live stream the camera images in the browser and output the text in real time. However, after much research, we decided that we didn't have the time to accomplish this, and eventually settled on a terminal program for the prototype. Keep in mind that this text to speech converter is NOT very accurate. Due to the time limitation, we had to cut a lot of edges, and make compromises where we probably wouldn't have liked to. The accuracy is somewhere between 35-55% based on the tests we conducted.

Accomplishments that we're proud of

We are really proud of what we've done, although it may not be where we wanted it to be. Given the time limitation (and more than 36 hours of looking at screen, downing caffeine and not sleeping), it's nowhere near perfect, but we are proud of it.

What we learned

We learned how to program and use a Raspberry Pi, play around the Clarifai API, and general machine learning.

What's next for Sign to Text Convertor

We don't know. We would like to improve on it so that the accuracy improves and is consistently more than at least 80%. Maybe in the future, who knows? ;)

Built With

Share this project: