Inspiration
Learning American Sign Language (ASL) is a highly visual and physical skill, yet most beginner resources rely on static images or dry textbook formats. We wanted to build something that made practicing ASL immersive, responsive, and genuinely fun. By blending educational mechanics with a gamified, medieval tavern setting, we aimed to create an experience where players are so focused on the gameplay that the learning happens naturally.
What it does
Echo Sign is an interactive, real-time ASL learning game. Players step into the shoes of a bartender in a bustling medieval tavern. As animated customers approach the counter, players must correctly perform specific ASL alphabet signs to serve them and keep the line moving. The game uses the player's webcam to track their hand movements in real-time, verifying their signs before fulfilling the customer's order.
How we built it
The core game engine and UI were built using Pygame, which handles the sprite animations, scroll menus, and background rendering. For the computer vision pipeline, we utilized OpenCV and MediaPipe to extract 21 3D hand landmarks. To ensure the game could recognize signs regardless of how close the player is to the camera, we engineered a scale-invariant math model. It calculates the relative Euclidean distance of all five fingertips from the wrist, normalizing the coordinates against the player's palm size. We also built custom coordinate-buffer logic to track moving signs, specifically the 'hook' motion of "J" and the 'zig-zag' of "Z". Finally, to maintain a smooth 60 FPS gameplay experience without the vision processing lagging the game, we separated the Pygame loop and the computer vision pipeline into different threads, using thread-safe queues to pass camera frames and success triggers back and forth.
Challenges we ran into
Getting a computer vision script and a game engine to play nicely together was our biggest hurdle. Initially, tying the camera feed and landmark detection to the same loop as the game rendering caused massive framerate drops and input lag. We had to completely restructure the architecture to use multithreading and queue management to stop the threads from blocking each other. Additionally, fine-tuning the mathematical thresholds for similar signs (like "P" and "K", or "G" and "D") required a lot of precise calibration to ensure the game felt responsive without accidentally triggering the wrong letter.
Accomplishments that we're proud of
We are incredibly proud of the multithreaded architecture. Getting the MediaPipe tracking to run continuously in the background while Pygame smoothly animates the customers and pulls the freshest camera frames via queue.get_nowait() feels like a massive technical win. We're also proud of implementing the motion-tracking buffers for dynamic signs like J and Z, rather than just sticking to static poses.
What we learned
We learned a massive amount about thread safety, concurrency, and managing asynchronous data streams in Python. On the computer vision side, we gained a deep understanding of spatial coordinates, Euclidean geometry in 3D space, and how to manipulate landmark data to make it scale-invariant.
What's next for Echo Sign
Our immediate next step is to upgrade the core detection logic from distance-based coordinates to vector-based angle tracking (measuring the angles between finger joints). This will make the detection completely immune to hand rotation and tilt. After that, we plan to expand the dictionary from the ASL alphabet to full words and phrases, and add PvP sign racing.
Log in or sign up for Devpost to join the conversation.