Sign Language Interpreter

Our CNN model recognizing the alphabet 'b'
Our CNN model recognizing different shapes
Our CNN model recognizing the alphabet 'a'

Inspiration

Our inspiration came from the recent boom in AI and utilizing it for the greater good. Creating a deep learning model for sign language interpretation is not just an exercise in technical skill; it's an opportunity to bridge worlds, to connect communities, and to give voice to the silence. Imagine a world where the barriers between the hearing and the non-hearing dissolve, where every gesture translates into words that everyone can understand. This is the world our project can help create.

What it does

Our project is a Sign Language Interpreter that uses a convolutional neural network to translate your hand actions into speech. It is imperative for people who want to understand what a sign language person is saying to them, effectively bridging the gap.

How we built it

We built it using the libraries PyTorch, TorchVision, and OpenCV. Our model includes a CNN layer for feature extraction on each frame of the live video, and that will be fed into an FC (Dense) layer for classification.

Challenges we ran into

The model training proved to be very challenging as it took a minimum of 4 hours for training. After each training, we fine-tuned the hyperparameters and re-trained it, which pushed the accuracy to 99.83% on the dataset. Dataset acquisition and preprocessing were also deemed a problem.

Accomplishments that we're proud of

We are proud of the accuracy of our model, which can accurately classify unique actions shown in sign without regard for who is doing it, effectively capturing the hand movements only.

What we learned

We learned a lot about the DNN (Deep Neural Net) libraries and how they work and how all that translates into a fully functioning model. Whenever we encountered an issue, we were able to fix that by going through the documentation of these libraries.

What's next for Sign Language Interpreter

The next step would be to "appify" (turn to an app) this Python project so that people can use it in their daily lives to understand the signs of a sign language person. Also, the model can be tweaked to understand complex hand gestures that involve a lot of movement using a CNN + LSTM combo for future use.