What it does
The application accepts Speech through a microphone and then displays videos that contain the translation American Sign Language (ASL) translation of what has been said.
What we learned
First and foremost, we learned how to string together applications that work in different languages to make one application that seamlessly offers the best features of all of them. We learned about the possibilities that Machine Learning can open us to, and we're very excited to use it in future projects as well. For some of us, this was our first time working with a Web App, and for others, it was our first time working with Flask. As a result we were able to add important tools to our skillset.
An unforeseen benefit was, in our effort to make the world a more equitable and connected place, we also picked up a bit of ASL ourselves!
One of our major challenges was to produce a well-organized idea with a clear goal. Even then, once we had an idea, finding the right resources to start our implementation was difficult. For example, working with Computer Vision libraries and several other tools was technically advanced for our skill level set. When we tried to train our own model to recognize ASL characters, none of our machines were powerful enough to train the model in a reasonable time frame. When we tried to find pre-trained models online that would be able to detect ASL and convert it into text, we struggled to find a model that worked as well as we needed it to.
Currently, the app only supports translation one word at a time. We're very close to enabling translation of sentences, but since this would still end up translating it word by word, we do plan on using better Natural Language Processing in the future to get better semantic information about the audio that is input. We also plan on adding multiple language support through Google Cloud's Translation API.
We also plan on training our own models in Python using Tensorflow to convert ASL input into text, that can then use Google Cloud's Text to Speech API to make this application a two-way communication line. With a bit more development, we can make this into a functioning mobile app as well.