Sign Shakti

Inspiration

There’s approximately 72 million deaf people in the world, and you may know someone in your life who actively uses sign language every day. Chances are, this person has learned ASL, or American Sign Language. While not an official census, estimates indicate there are about half a million to 2 million native ASL users. But did you know that there’s a whopping 3-12 times as many people who use the Indo-Pakistani Sign Language? In order to address this market and increase accessibility for ISL speakers, we decided to create an ML model to interpret ISL.

What it does

Taking an image from a webcam, it feeds it to an ML model, which interprets the hand gesture, and then puts an interpretation over the webcam feed.

How we built it

We prepared the data, trained the model, and deployed the model on a frontend. We found the data with Kaggle user Pathikreet created his own set of images of him performing the same gesture. Since a model needs a variety of data to train effectively, we mixed it up by augmenting these images and transforming them to simulate different situations, like lighting conditions and skin tone. From here, we placed these hands randomly on other images, which we eventually replaced with actual people using a dataset of youtube frames.

Afterwards, we trained a base model called MobileNet V2, which was a model capable of object detection on this synthetic data. Once we had a model, we converted it to a TensorflowJs json file that reactjs could interact with, and loaded it into a frontend.

Challenges we ran into

We ran into a lot of issues loading the ML model json into the react frontend... it refused to recognize it as json, then it said it wasn't a proper json, then it would only load null values, and there were a lot of other hiccups, which were especially nerve-racking since it was so close to the deadline.