Teaching American Sign Language with Transfer Learning
Currently over 5% of the global population has some type of disabling hearing loss. In order to compensate this, millions of people use American Sign Language (ASL) to better communicate with others. However, communicating with the general public is still a struggle as most of them are unaware of such a language. So our goal was to help introduce the larger public to ASL to help better integrate those with disabling hearing loss.
What it does
"Teaching ASL using Transfer Learning" uses a neural network in the browser to detect and identify different ASL gestures. It does this by comparing it to images of an ASL gesture performed by the user, so that it can provide useful engaging feedback to the user to see if they are able to imitate the right gesture. By using these techniques we built a small interactive text based adventure that uses different gestures to progress to create a fun and interactive experience for the user, promoting the use of ASL.
How we built it
Challenges we ran into
Over the course of this project we ran into a few challenges mostly around asynchronous js and getting our heads around the implementation of the neural net in tensorflow.js .
Accomplishments that we're proud of
During this project we had a few accomplishments that we're proud of. We were able to train a working neural network that was able to identify ASL gestures using very little data due to transfer learning all in a web browser! This means it is lightweight, portable and surprisingly powerful.
What we learned
What's next for "Teaching ASL using Transfer Learning"
Although we initially used the alphabet as a means of communication, in future we plan on integrating signs representing whole words and continuously updating our library of signs so eventually anyone can pick up our game and better communicate with those with disabling hearing loss.