Show-And-Tell

Inspiration

We were interested in exploring machine learning and wanted to dive into learning about computer vision and neural networks through the creation of a sign language interpreter.

What it does

Reads sign language from the webcam, feeds the image into a neural network, predicts a sign, then outputs the predicted sign back to the user.

How we built it

Using a dataset of 80000+ images we trained our own neural network model using Google Colab, tensorflow for building the model, tensorflowjs and React for the web components.

Challenges we ran into

Creating a neural network model and finding out how to actually implement it. Model training require a lot of time and we encountered multiple times that Google colab timed out when the model training is half way through, forcing us to restart the training process multiple times. As a result, due to time limitations, we had to decrease the number of epochs which significantly decreased the accuracy of the model.

Accomplishments that we're proud of

Built and trained a deep neural network model from a dataset! (That was accurate in test too!)

What we learned

Tensorflow, machine learning and other neural network technologies have an insane skill ceiling and mastering these skills opens many opportunities for potential projects. Many neural networks can be made to solve a variety of problems.