SignSense

Inspiration

A close friend of one of our developers was acquainted with the struggles of communicating with deaf people such as his freshman year roommate due to the lack of adequate available on the internet. The main reason why current resources available to us are lackluster are because they either do not support real-time translation or they don't truly understand sign grammar and the formulation of sentences from translated sign words.

What it does

SignSense intends on servicing real time sign translation and forming a group of words into the most likely sentence they could produce. This would allow for a fully functional conversation between a person who does not know ASL with a deaf person. Additionally, we have currently implementing a learning section into our product which would allow a deaf/non-deaf person to learn the language daily similar to learning language on Duolingo.

How we built it

For the backend, we used Google's MediaPipe to map the coordinates of hand movements in real time videos that are captured with the help of the cv2 libraries, PyTorch to build our model, and some basic libraries such as Numpy and others for logic and calculations. For the frontend, we used React and Tailwind CSS. For building our dataset, we accessed numerous preexisting datasets available online and we used yt-dlp and FFmpeg to extract extra training videos.

Challenges we ran into

There were several challenges we ran into. First of all, we weren't able to get access to a lot of the key datasets available online due to them requiring permission and we resorted to using a large compilations of smaller datasets which tampered our model. Additionally, we ran into issues with the logic for our model as far as what it was prioritizing during its training. Specifically, we were challenged with deciding whether the model should focus on velocity, coordinates, or angles of each finger.

Accomplishments that we're proud of

We're proud of the way we planned and structured the focus of our project and our research into what the deaf community really requires out of a ASL app. We also feel accomplished as to the fact that we are in the right step towards providing a similar form of DuoLingo but in a ASL version, which upon research we found out was in very high demand.

What we learned

We learned about the complexities of training a machine learning with somewhat limited data available to train with. We also learned a lot more about the nuances of sign language and got a greater appreciate for it as a whole as we realized the language is a lot more complex than it appears.

What's next for SignSense

We want to continue to build our library of words and our datasets to make our model as accurate as possible. We also want to create some sort of ASL Duolingo to teach both deaf/non-deaf people as it's in high demand. We hope that we can develop this product in close contact with the deaf community around the world so that it provides a true, meaningful impact to them.