We wanted to create a piece of software that would improve communication and accessibility.
What it does
pySign uses a convolutional neural network to classify the letters of the American Sign Language alphabet, and uses a web interface to provide live translation via webcam.
How we built it
We used pytorch to train the neural network, flask to make a small python server that could handle requests from the web frontend, and html/css/js to make a small web app that communicates with the server.
Challenges we ran into
We found certain pretrained models that we were attempting to use to simplify training for our CNN more difficult to implement, and we found that some models either trained far too slowly to be useful or simply could not be trained to be more effective.
Accomplishments that we're proud of
We're proud of training a neural network that can identify inputs with reasonable accuracy, and of being able to create a finished product in such little time.
What we learned
We learned a great deal about the computational and theoretical complexity of machine learning. We also learned about connecting programs to web applications.
What's next for pySign
pySign's future involves finding a more effective model to use for identifying ASL and a more permanent website to be served from. In the further future, we might look to teach pySign to recognize live video, and to teach it more signs that just the ASL alphabet.