SignSync

Inspiration

One of the main ways that friends bond is through playing games and doing other fun activities together, but that isn't always a possibility for everyone. We were inspired by Meta's challenge to connect people together to create an app that enables anyone, hearing or deaf, to have fun and bond over ASL challenges and wacky visual results.

What it does

Our game lets two players join a room and work together to sign words and collect points. After the completion of every word pair, there is an AI-generated image of the combination of the two words for players to see and enjoy. Once the players have completed enough word pairs to reach a point threshold, they win! They can see all their generated images at the end to save and are offered to play again. If they chose to hide their web cams from each other at the start, then they have a chance to guess the other person's words based on the generated images as a interesting, secret game at the end!

How we built it

We used Node.js as the backend, browser-based hand-sign recognition, and multiplayer gameplay to bring this game to life. We used Meta Muse API to generate the combination images and all advice in the builtin tutorials and tutor chat bot to help users learn ASL and have a good experience playing the game. For hand tracking, we used Google’s public MediaPipe hand-landmarker resources. The app is designed to be deployed on Render to connect the game to a website, so that multiple users can join on separate devices.

Challenges we ran into

The hand tracking and the camera created a few challenges since the camera screen glitched depending on the timer, the hand tracking was flipped due to our cameras having flipped footage, and we had to implement ways to make sure the letters are separated while the user is playing, since being exceptionally fast can make the tracking slightly inaccurate.

Accomplishments that we're proud of

We created the game accounting for users of all ASL skill levels, and we are proud that the game is open for everyone and inclusive. We are proud that we used computer vision to accurately analyze which sign someone is doing and that the game uses AI agents to customize tutorials and advice for every player.

What we learned

We learned a lot more about how computer vision and AI integration work, since they are vital parts of the project that allow us to analyze ASL signing and create a game around them. The project also taught us how to uphold inclusivity, since we were trying to implement more functions to bridge ASL-ability gaps and create an environment for everyone to enjoy.

What's next for ASL Game

In the future, we hope to let users have the option to match games, so that people can play with strangers and connect, forming bonds all around the globe. To do this, we would have to add a feature where, instead of joining a room via a room code, the website manually matches people together and sends them to a room. We hope to accomplish this and make our game connect even more people! Additionally, we can implement words into the game where players can create wacky sentences and generate those visuals for them as well!