Catiator

Interactive VR Game to Learn and Improve American Sign Language Skills through Gesture Recognition.

Comment 3

What Does "Catiator" Mean?

Cat·i·a·tor (noun): Cat + Gladiator! In other words, a cat wearing a gladiator helmet 🐱

What It Does

Catiator is an educational VR game that lets players battle gladiator cats by learning and practicing American Sign Language. Using finger tracking, players gesture corresponding letters on the kittens to fight them. In order to survive waves of fierce but cuddly warriors, players need to leverage quick memory recall. If too many catiators reach the player, it's game over (and way too hard to focus with so many chonky cats around)!

Inspiration

There are approximately 36 million hard of hearing and deaf individuals live in the United States, and many of them use American Sign Language (ASL). By learning ASL, you'd be able to communicate with 17% more of the US population. For each person who is hard of hearing or deaf, there are many loved ones who hope to have the means to communicate effectively with them.

"Signs are to eyes as words are to ears."

As avid typing game enthusiasts who have greatly improved typing speeds (TypeRacer, Typing of the Dead), we wondered if we could create a similar game to improve the level of understanding of common ASL terms by the general populace. Through our Roman Vaporwave cat-gladiator-themed game, we hope to instill a low barrier and fun alternative to learning American Sign Language.

Features

1. Multi-mode gameplay. Learn the ASL alphabet in bite sized Duolingo-style lessons before moving on to "play mode" to play the game! Our in-app training allows you to reinforce your learning, and practice your newly-learned skills.

2. Customized and more intuitive learning. Using the debug mode, users can define their own signs in Catiator to practice and quiz on. Like Quizlet flash cards, creating your own gestures allows you to customize your learning within the game. In addition to this, being able to see a 3D model of the sign you're trying to learn gives you a much better picture on how to replicate it compared to a 2D image of the sign.

How We Built It

  • VR: Oculus Quest, Unity3D, C#
  • 3D Modeling & Animation: Autodesk Maya, Adobe Photoshop, Unity3D
  • UX & UI: Figma, Unity2D, Unity3D
  • Graphic Design: Adobe Photoshop, Procreate

Challenges We Ran Into

1. Limitations in gesture recognition. Similar gestures that involve crossing fingers (ASL letters M vs. N) were limited by Oculus' finger tracking system in differential recognition. Accuracy in finger tracking will continue to improve, and we're excited to see the capabilities that could bring to our game.

2. Differences in hardware. Three out of four of our team members either own a PC with a graphics card or an Oculus headset. Since both are necessary to debug live in Unity, the differences in hardware made it difficult for us to initially get set up by downloading the necessary packages and get our software versions in sync.

3. Lack of face tracking. ASL requires signers to make facial expressions while signing which we unfortunately cannot track with current hardware. The Tobii headset, as well as Valve's next VR headset both plan to include eye tracking so with the increased focus on facial tracking in future VR headsets we would better be able to judge signs from users.

Accomplishments We're Proud Of

We're very proud of successfully integrating multiple artistic visions into one project. From Ryan's idea of including chonky cats to Mitchell's idea of a learning game to Nancy's vaporwave aesthetics to Jieying's concept art, we're so proud to see our game come together both aesthetically and conceptually. Also super proud of all the ASL we learned as a team in order to survive in Catiator, and for being a proud member of OhYay's table1.

What We Learned

Each member of the team utilized challenging technology, and as a result learned a lot about Unity during the last 36 hours! We learned how to design, train and test a hand recognition system in Unity and build 3D models and UI elements in VR.

This project really helped us have a better understanding of many of the capabilities within Oculus, and in utilizing hand tracking to interpret gestures to use in an educational setting. We learned so much through this project and from each other, and had a really great time working as a team!

Next Steps

  • Create more lessons for users
  • Fix keyboard issues so users can define gestures without debug/using the editor
  • Multi-hand gesture support
  • Additional mini games for users to practice ASL

Install Instructions

To download, use password "Treehacks" on https://trisol.itch.io/catiators, because this is an Oculus Quest application you must sideload the APK using Sidequest or the Oculus Developer App.

Project Credits/Citations

Built With

+ 23 more
Share this project:

Updates