ASL Translator

Initial UI concept
First screen (camera background)
Second screen (camera background)

Inspiration

We wanted to tackle a massive problem that the deaf community faces in a world post-COVID, where Zoom meetings and remote jobs are more relevant and frequent than ever. An audibly impaired individual faces a lag in communication through the screen when it comes to ASL. If there is a translator on-site, it takes a while for their message to get across. We were inspired by leap motion and its accurate hand motion tracking sensors. We learned that video conference calls is a struggle that deaf people people face. We wanted to create some software that can help provide accurate tracking for hand motions that eventually can take ASL through the live web cam feed and provide a speech translator.

What it does

To solve this, we wanted to create an application that recognizes ASL, tracks hand motions and translates to text, and then outputs speech based off that text.

How we built it

We utilized a Tensor Flow Lite model that used 190GB of ASL data and deployed it onto Android Studio where we developed our app. Within our app we had a text to speech program that would output the input text into speech. We also had a frontend component where we designed two screens for our application.

Challenges we ran into

We ran into several challenges while trying to complete this project. The first, and biggest obstacle, was finding the right machine learning model that worked for us. We also had to convert Java code to Kotlin to accommodate for the version of Android Studio that we were working with. Considering the scope of our project, we realized that the time constraint is too little. We also did not have enough time to mirror leap motion's sensor with our computer's webcam.

Accomplishments that we're proud of

We are proud of being able to create a framework for the app, deploy the model, and work together towards a common goal. We're also proud of our ambition to work on this project’s idea considering its complexity.

What we learned

We learned how to utilize a Tensor Flow Lite model, and how to use and connect Leap Motion’s sensor.

What's next for ASL Translator

We are going to completely revamp ASL Translator, probably to a new platform, and re-do all of our programs so they actually integrate together and work.