Speechless

Inspiration

Accessibility is important. We realized the issue of accessibility in video calls when Gerry noticed a coworker video calling her husband everyday at lunch - in sign language. Today, around one million people use American Sign Language as their main way to communicate, yet the majority of the world cannot use or understand it. Gerry noticed this when his coworker seemed to have difficulty signing phrases to her husband through video call. Thus, we aimed to solve this communication problem by building a video chatting platform where users are able to call their loved ones and chat using sign language. The platform translates words in real time, displaying them as live subtitles in the video screen during the call so that users are able to easily communicate despite the restrictions of their accessibility needs.

What it does

Speechless is a video calling platform that translates sign language in real time in the form of subtitles during a call. Currently, it supports the reading of the 26 letters of the English alphabet as well as a "space" character for separating words.

How we built it

Speechless uses a pre-trained machine learning model that recognizes the letters of the alphabet in ASL, then appends the signed characters to a string which is outputted as subtitles when a space character is recognized. The OpenTok API allows different users to join a video chat with a unique token to enter a chat room and share video.

Challenges we ran into

We used the OpenCV API to allow the model to identify ASL characters from a live video feed from the webcam . At the same time, we needed to access the webcam using the OpenTok API, which made it impossible to display subtitles at the same time as a video call. We handled this by adding in screen sharing features during a video call, which allowed subtitles to be displayed while video calling in real time.

Accomplishments that we're proud of

Each of us used new frameworks and technologies we had never used before, such as Tensorflow, Keras, OpenCV, OpenTok API and Bootstrap
Managing to build something while learning simultaneously in a short period of time.
Being able to overcome huge issues and problems as a team.

What we learned

We learned how to use different components of our project and put it all together We all learned new technologies and despite our lack of knowledge we came a long way and managed to put together a product with a newly learned skillset.

What's next for Speechless

Refinement with faster character recognition
Adding phrases to be recognized as many ASL words can be signed in a single gesture, makes signing and translating words faster and more efficient