SignFlow

Inspiration

We wanted to build this to help deaf people communicate with other people properly instead of relying on human translators being by their side. Additionally, we thought using computer vision to detect ASL would be a good test for KrinosAI if we could make a system that provides precise labels in real time.

What it does

It tracks your hand and provides a confidence value on what you are signing to provide real-time ASL translation.

How we built it

We built the frontend using html, css, and vanilla javascript. We decided to make our UI similar to an infographic where we wanted our app to look more educational to the users, with all the information on one page.

To build the predictive model, we use an LSTM (Long Short-Term Memory) network to predict more complex terms that involve movement, such as translating "I love you". For the static letters, we use an MLP(Multilayer Perceptron) to detect single letters. To track the hands, we used mediapipe, which gave each hand 11 nodes that track the position.

Challenges we ran into

One of the problems that we ran into was with mediapipe as it required using older versions of python which took a while to debug and delayed us a bit. Additionally when trying to get the LSTM model to try and predict more complex signs we had difficulty getting mediapipe to track both hands. Another problem we encountered with the more complex signs was differentiation between similar signs like “please” and “my”. Ultimately we decided this problem would be too difficult to fix within the given time frame so we only used signs that were easy to differentiate. Finally, we also encountered a problem with setting up a server to run the website.

Accomplishments that we're proud of

The accomplishment we are most proud of is getting our model to predict some complex signs. We were unsure if we would be able to get something like this to work as the movement used in these signs was different from the still letters we were used to. So we feel accomplishing this proves our model could work in a more realistic environment and work better for real time labeling.

What we learned

During this process we learned about LSTMs and a basic understanding of how to code them. Additionally we learned how to use mediapipe when gathering data for the signs.

What's next for SignFlow

If we had more time we would add more phrases to the model. Additionally, we want to try and add back signs that look similar. We hope that part of the reason that our model had difficulty differentiating between these signs was because we lacked enough data for our model to accurately spot the differences between the two.