SignSprites

SignSprites is a 2D adventure that gamifies ASL learning. Players use a webcam and real-time AI to validate hand signs, turning alphabet mastery into an immersive, interactive quest.

Comment

Inspiration

Inspired by the soot sprites from Spirited Away, players guide a Ghibli-style sprite across the screen, collecting paper stars. Along the way, the sprite encounters obstacles that require the player to sign a specific ASL letter to pass.

What it does

  • Real-time Tracking: The game activates the webcam and tracks the player's hand in real-time.
  • Smart Detection: It verifies if the player is signing the correct letter.

How we built it

  • Frontend: Built with React, TypeScript, and Vite for a component-based UI, with hand-drawn elements and backgrounds.
  • Backend: Built with Firebase for quick user authentication.
  • Computer Vision: We implemented @mediapipe/tasks-vision to extract 21 precise 3D hand landmarks from the webcam feed.
  • Machine Learning: We flattened the MediaPipe coordinate data into a 63-point array and used the fingerpose library to identify static ASL gestures.

Challenges we ran into

We also did try to integrate Gemini for real-time feedback as users did their signage, but unfortunately were unable to finish due to time constraints and it would be something we’d like to explore in the future. We were also using an existing neural network called ml5.js to detect different letters in ASL. However, the training process was very time and resource intensive, using almost 3,000 original photos and videos in the span of 5 hours and was still not completed, due to which we had to replace ml5.js with fingerpose.

Accomplishments that we're proud of

We are proud that we were able to get our computer vision software to work. We were able to merge the software with our fingerpose file which allowed the game to compare the user’s actions with how the hands are supposed to look.

What we learned

We were able to build a basic foundation in computer vision as well as get introduced to agentic AI

What's next for SignSprites

We aim to advance SignSprites to account for the hand movements as well, since a lot of ASL depends on hand movements. We also aim to be able to teach students how to sign words and sentences in the future, which will take a lot more training and code for defining each word and its meanings.

Built With

Share this project:

Updates