ASL Arena

Title Screen
Opening Scene
Learning the Alphabets!

Inspiration

ASL is the American Sign Language that is used widely by the hearing impaired individuals. One of the devs of this project is a tutor, and while meeting with their student, they came to know how the student learned ASL to communicate with their deaf grandparent. The student also playfully ignored the dev while they studied until they used, or made an effort to use ASL. The dev realized that soon, under the stress of communicating something, they had learned a decent bit of the language.

Therefore, using their want to create a pixel game, and their friend's willingness to help them with this journey, they set off to develop ASL Arena!

What it does

ASL Arena is a multiplayer wizard role-playing game inspired by "Mage Arena". The player has to learn some basic spells (ASL alphabets), and use those to defeat their friend in a PVP match! The game serves as a fun and engaging way to teach ASL to people, since the person playing can do multiplayer battles with their friends, which makes it fun and competitive!

The game play loop is as follows:

Learn in the dojo. Each player starts in their own pixel-art village and walks into a dojo. A board shows one hand-sign illustration at a time (L, W, I, U, K, P), and you learn it by making that sign at your webcam. Signs you've learned can be clicked to practice again.
Cross the bridge. NPC wizards on the bridge give tips as you walk to the arena.
Duel. In the arena, a letter you've learned appears over your head. Sign it at your camera and your wizard casts a fireball.
- Sign quality sets the power: a sloppy sign throws a weak fireball and a clean one throws a big one. Big hits shake the screen.
- Each sign has a cooldown, so rushing into the arena knowing one letter leaves you with very few attacks.
- Mid-fight pickups teach new signs.
- Hesitate too long and a card pops up showing the correct hand shape.
Don't keep your opponent waiting. When one player enters the arena, the other has 10 seconds before guards come and escort them in.
…and sometimes, a goose. About 30% of strong attacks come out as a crazed goose that homes in on your opponent and honks nonstop until it lands.

There's also a Quick Duel mode against a practice dummy, so anyone can pick it up and play in under 30 seconds.

How we built it

We started by feeding a dataset of images of hands signing the ASL alphabet into Google's Teachable Machine. From there we developed a detector that tracked hands as well as if said hand is signing an ASL letter on live video input. We then used Godot as our game engine to act as the base for making the actual game mechanics. ALL the designs and animations of the characters and environment were created by hand in Aseprite. From there we combined the ASL detection with the game on Godot. After refining the detection system and PVP mechanics, ASL Arena is ready to play!

Challenges we ran into

Since we built the game in two of its separate parts, the first being the hand tracking and the second being the game itself, we had to brainstorm for a while to integrate the two halves together. The hand tracking model also required extensive training in order to properly classify letters, which was tackled by using a Kaggle dataset.

Accomplishments that we're proud of

We are proud of the fact that we were able to develop an entire game from scratch within the hackathon deadline. Moreover, we are proud that what we built makes learning something as impactful as ASL fun, easy and collaborative! Shrikant is proud that they will be able to share this with their student :)

What we learned

Through this project Kayla learned how to use Google's Teachable Machine to train a model on given datasets, alongside improving the detection system through small tweaks and gradual refinements. This was also a great chance to learn and experience using Git in a setting outside of the traditional classroom.

Shrikant:

The wonderful world of Aseprite utilities! There is so much you can do with them.
Using Godot to make games

What's next for ASL Arena

ASL Arena only features a limited selection of signs to learn and use in combat. The next steps would naturally be to implement the entire ASL alphabet so that all the letters can be learned in one game. The detection tended to be faulty when trying to decipher between very similar signs such as M, N, and T. Using a more expansive data set would refine and likely improve the accuracy of detection and identification of signs.