Gesture Play

Inspiration

Back in my college days, my professor always used to say, "Focus on how you can solve real-world problems with software." That stuck with me. After working as an XR developer for years, I kept thinking I should make something for the education or medical field. Recently, when we saw the new Hand Tracking updates in Unity, we thought, "Why not build a Sign Language trainer?" The main goal is to contribute something useful for people who want to learn non-verbal communication and understand the emotions of those who cannot speak.

What it does

Gesture Play is a VR app that helps you learn American Sign Language (ASL) without touching any controllers—you just use your real hands. It has two modes: a "Practice Mode" to learn alphabets comfortably, and a "Game Mode" (Infinite style) where signs come at you like a rhythm game, and you have to sign correctly to score points.

How we built it

Initially, we thought to go with Unity OpenXR and the standard Interaction Toolkit. We tried a lot, but honestly, the results were not up to the mark. After a lot of trial and error and R&D, we shifted to the Meta XR Interaction SDK. That changed everything. We captured the hand micro-gestures and mapped them to the logic. We started with ASL because there is good data available for it.

Challenges we ran into

The main issue we faced was tracking complex hand signs. Current VR hardware still struggles when fingers overlap (self-occlusion). It was really tough to get the detection right for some specific alphabets. We had to tweak the code a lot to handle these limitations so the user doesn't get frustrated.

Accomplishments that we're proud of

We are really proud that we made a completely controller-free experience. Seeing the hand tracking work smoothly in a game environment feels great. We successfully implemented the core American Sign Language gestures, and the gamification makes learning actually fun rather than boring.

What we learned

We learned that hand tracking technology has improved a lot, but there is still scope for optimization. We also realized that we can build many more apps in this area for social good.

What's next for Gesture Play

As I mentioned, there are still some hardware limitations for complex shapes. We are waiting for some updates on that front. But in the future version, we are planning to add many more languages like British (BANZSL), Chinese (CSL), Japanese (JSL), Arabic, and Spanish Sign Languages. We want to make this a global platform for everyone.