Sign-to-text

Training video (ASL)

Inspiration

According to the World Federation of the Deaf, there are around 70 million people worldwide whose first language is sign language. Gallaudet University estimates anywhere from 500 000 to two million ASL users in the US alone, yet, not many people know how to sign. We wanted to bridge the gap between those fluent in ASL with those who are not, Google Translate style.

What it does

Given a video of an ASL word, the application is able to detect and transcribe the word to someone who knows nothing about ASL.

How we built it

Everything is hosted on Google Cloud Platform. We pulled a ton of videos from the American Sign Language Lexicon Video Dataset, hosted by Boston University to train our model. We used opencv for Python to process and mask the input frames and used Keras to do some machine learning to classify the words.

Challenges we ran into

OpenCV (surprise, surprise) was a pain in the butt to work with. Some of the especially challenging parts of this hack included trying to detect hands which could be in front of the face or forearms. In either of these cases, we couldn't simply mask out colours, since the background is basically the same colour as the hand.

Another thing that we had issues with was the machine learning component of the hack. Some of us have done smaller machine learning projects before, but nothing quite of this scale. The convolutional autoencoders were new to all of us, and finding a way to model and classify words was another challenge altogether. Luckily we got some help from some awesome people that helped us push through these challenges and succeed!

Accomplishments that we're proud of

How much we accomplished this hackathon
How far we've come
Our resolve not to give up when things seemed impossible at times

What we learned

Convolutional autoencoders are hard
We're pretty good at multithreading
ASL recognition is a _ really _ hard task
No matter how many times we say we will never hack with opencv again, we will

What's next for Sign-to-text

Ideally, we would keep going and make this the future of ASL-to-English translation. Our goal is that one day you would be able to get a quick clip of someone signing ASL to you and it would be transcribed. We feel that this could be like Google Translate for ASL!