Inspiration
Most translators in the market are one-way communicators for the deaf-mute community, as they only take text as input and output gestures, but not the other way around. Our project is to build a real-time sign language (i.e., ASL) translator, which takes live gestures and translates them simultaneously.
What it does
Unlike common sign language recognition projects, which inputs the sign gesture as a static image and labels image, our model after training can take a series of live gestures via camera and translate at the same time. That huge step from ‘static prediction’ into ‘real-time translation’ makes us unique.
How we built it
To build a real-time sign language translator, our first step is to build a real-time sign detection model, and to do that we combine Tensorflow Object Detection and OpenCV. We first capture and label images, then transform images into a Record file (through a python script by Nicholas Renotte) which can then be used for faster image training, and we train the model, named ssd_mobilenet, from Tensorflow Model Zoo, then we define detect function based on the model we trained, and connect to our webcam, apply the detect function while the webcam is turned on. Based on these, we have built a real-time detection model that can translate signs like ‘thank you’ and ‘thumbs up’ with high accuracy (please click on our GitHub page for more details https://github.com/YuHuang0525/Real-Time-Gesture-Detection-Translation ). We also find video clips of single words on http://vlm1.uta.edu/~athitsos/asl_lexicon/ for later training, and once our model achieves high accuracy for most single signs, we will change from individual words to sentences by training the model with sequential images.
Challenges we ran into
Challenges that we have encountered and we foresee are 1) labeling images will be time-consuming, and we later need help from freelancer; 2) generalization of the model (i.e. the model trained by my images does not function the same way when detecting info from my partner (sign - ‘thumbs up’ done by me is identified as 0.93 close to the ‘ground truth’, but the exact same sign done by my partner is identified only as 0.83). We think that is because the details in the image are emphasized by the training process, i.e skin color, thus for some signs we need to record clips ourselves.
Accomplishments that we're proud of
We are very proud that the resulting accuracy for signs like 'thank you' has reached 0.98 and we are very excited about our next steps of training sentences in ASL.
What we learned
Through the project, we learned about how we build a real-time detect model and how to connect to the model we build in real-time through OpenCV. Moreover, we learned how to train images faster by using our GPU.
What's next for Real-Time Gesture Detection & Translation
We also find video clips of single words on http://vlm1.uta.edu/~athitsos/asl_lexicon/ for later training, and once our model achieves high accuracy for most single signs, we will change from individual words to sentences by training the model with sequential images. Datasets of simple sentences in ASL are found on https://www.lifeprint.com/, and we need to decompose clips into sequential images with PS. The resulting model aims at projecting a sequence of signs onto corresponding English sentences in real-time.
Moreover, once established, our model allows gestures personalization, which allows users to easily communicate beyond ASL and with any gestures they prefer. With that special feature, our model makes communication between different deaf-mute communities more convenient (i.e. American vs. Chinese); they don’t have to learn different sign gestures, as long as they label their gestures correctly when adding that gesture to the model the output would be the same text label for understanding.