VASLR

Inspiration

We were inspired by digital language educative applications such as Duolingo and the interactivity of Virtual Reality applications, such as Noun Town, a VR language application dedicated to teaching spoken languages in a third-dimensional digital space.

What it does

We constantly scan the user's hands to recognize hand gestures and hand poses, which are used to compare against stored hand gestures/poses. And, within a given margin of error, it will detect which ASL sign the user is putting up. In this specific project, the user is situated in a cafe, where he will learn how to order a coffee or tea using ASL by conversing with an NPC barista.

How we built it

This was built largely with Oculus' built-in hand tracking software and Unity to create the 3D interactive environment/visualizer for the hands.

Challenges we ran into

Figuring out how to convert raw, hand joint data from the hand tracker into parsed hand gestures. Additionally, we also had difficulty differentiating between similar hand gestures, like the ASL sign for a "U" and "V".

Accomplishments that we're proud of

Getting the hands to actually visualize on the Unity space as well as getting the scripts to determine which hand sign is being put up were the two main accomplishments we are particularly proud of.

What we learned

Managing workloads in a team-based setting can be quite challenging since the work may not be as clearly defined as it may seem. Additionally, translating raw hand joint data from the VR set proved quite difficult since it required working with Meta's built-in software.