About the project Be sure to write what inspired you, what you learned, how you built your project, and the challenges you faced. Format your story in Markdown.

Inspiration

There are people around us who may have trouble communicating verbally. My eight-year-old niece once told me that one of her classmates, Alice, had a hard time communicating with other kids using spoken language. Therefore, the teacher taught Sign Language to all of her classmates so that Alice could have a way to express herself while the other kids could also better understand her needs and develop a friendship with Alice.

This is one of the sweetest stories I've ever heard of, and it also makes me realize that there are people around us who need to leverage ASL (American Sign Language) in order to get themselves across, such as the Deaf community, people with hearing loss, autistic children, etc..

This inspires me to think about what I can do to help normal people establish more effective communication with these communities -- Sign Language Gloves is my first step towards this journey.

What it does

This Sign Language Gloves filter translates English speech to ASL (American Sign Language).

With this filter, a set of predefined keywords (mom, dad, grandpa, sister, brother, dear, baby, etc.) and common phrases ('I love you,' 'I miss you,' 'thank you,' etc.) can be captured via voice recognition and converted to 3D signs that shown with virtual gloves.

How we built it

We created the ASL animations with Blender.

We leveraged the VoiceML module provided by Lens Studio for voice recognition and 3D pose tracking to attach the gloves to people.

Challenges we ran into

Generating ASL animations takes a longer time than we expected. Initially, we plan to use webcam-based motion capturing to generate the animations. However, the quality is not good enough for the application. Finally, we cracked it by combining the mocap results with manual correction. The voice control pipeline is difficult to use at the beginning. But fortunately, the Len Studio example provides a great template for us to follow and we are able to manage the pipeline over time.

Accomplishments that we're proud of

The magic gloves work! My niece tried the lens and she is pretty happy with it! She wants to show it to Alice.

It’s always exciting to build something with AR technology, and more importantly, to serve a goal which is beyond technology. We are glad that every individual can have easy access to this filter and leverage it as a different way of communication.

What we learned

There are always unknown challenges and difficulties on this journey, but we will try our best to overcome them and chase our goal relentlessly.

What's next for Sign Language Gloves

We are planning to support more words and sentences for different contexts. If possible, we are interested in loading the ASL signs from cloud APIs so that the filter size will not constrain the functions. The delay of the current transcription is still longer than expected. We are planning to provide real-time interpretation and continuous phrases.

Built With

Share this project:

Updates