HandsUp

HandsUp: Learn to sign through play and real-world scenarios.
Airport: Learn using basic sign on airport
DIFFERENT SCENARIO CHOSEN FOR LANDING PAGE
RUSH HOUR
Sign Competition
AI Guess: Test your hand shapes with real-time, on-device AI recognition.

Inspiration

Communication is one of the most fundamental building blocks of human connect. Yet not everyone has equal access to it.

Sign-language users often live in a world designed primarily around spoken language. Much of the responsibility for overcoming the communication barrier falls on the hard-of-hearing and Deaf communities, they are frequently expected to learn spoken or written communication.

However, communication should be a two-way effort.

Instead of asking only sign-language users to adapt to speaking and hearing world, we asked: What if we could encourage more people to learn some basic hand signs to communicate with them?

Our project aims to make the first step into sign language simple, accessible and fun through short, scenario-based mini games.

Rather than teaching an entire language at once, we focus on small pieces of communication that people can immediately understand and memorise by practicing it.

The missing block is not on the Deaf side alone. Sometimes, it is our willingness and ability to meet them halfway.

What it does

HandsUp is a free, no-login, no membership required sign language learning game designed to introduce people to useful signs through everyday scenarios.

Users able to choose a real-world scenario, such as:

Bus/ Public Transport
Airport
And more scenario as the platform grows

Each scenario aims to teach the user at least three useful signs that could help them communicate in that environment.

Instead of the traditional lessons, users learn through interactive mini-games. For example, in the airport scenario, a player might learn signs associated with simple interactions such as check in or thanking someone.

The goal is not to make someone fluent in sign language after playing a game. Instead, we want to make learning the first few signs approachable enough that anyone is willing to start.

How it works

How it Works The game is designed around a simple learning loop: Choose a scenario → Learn signs → Practise → Play → Receive feedback

Choose a Scenario:

Users select an everyday situation they want to explore This gives the sign context rather than presenting users with an overwhelming vocabulary list

Learn

The game will introduce a small set of signs relevant to that scenario By keeping each lesson short, users can quickly learn something useful without committing to a full course

Practise Through Games

Users reinforce what they learned through mini-games Different game modes can test recognition, memory and physical signing

Bonus game

Users can explore the bonus games that go beyond the scenario-based learning experience These games focus on making sign-language practice more fun, social and competitive, as it allows users to play with friends, for example our Head To Head lets two players compete against each other using the signs they have learned

Instant Feedback

Players receive immediate feedback and can retry gestures, encouraging experimentation and learning through repetition.

How we built it

We build our platform using React19, TypeScript, Vite and Google MediaPipe Gesture Recognizer with zero backend. All gesture recognition happens directly in the browser, meaning there is no account, no server processing and no video upload. Camera frames never leave the user’s device - they process locally and immediately discard.

Real Time Gesture Recognition

Google MediaPipe is used to track 21 hand landmarks and recognise gestures in real time. However, we quickly discovered that relying on a pre-trained classifier was not reliable enough for our games. For example, MediaPipe can confuse Thumb_up and Closed_Fist as the two gestures have similar hand shapes. In order to improve the recognition, we built our own geometric scoring system that uses the raw hand landmarks as a second source of evidence.

Instead of making decisions from a single camera frame, we also calculate recognition confidence across an 800 ms rolling window. This helps to reduce unstable predictions and makes games more consistent.

Recognising Movement, Not just Hand Shapes

Some of our mini-games require more than recognising a static gesture.

For example, in our 6-7 motion game, the gesture is defined by movement, meaning no single camera frame contains enough information to recognise it. Therefore, we developed a state-machine based motion detector that tracks both hands across multiple frames and detects when required movement has been completed within the frame.

This gave us two complementary recognition systems: one for static hand shapes and another for movement-based gestures.

Built for Reliability

We separated our recognition and game logic from the React interface so that the core behaviour could be tested independently from the camera and UI.

This allowed us to replay complete game rounds frame-by-frame during testing without requiring a webcam, browser DOM or real-time timers. By the end of the hackathon, we had XX passing unit tests across XX TypeScript/TSX files.

We also designed the game to fail gracefully. If GPU-based recognition is unavailable, the system will fall back to CPU processing. If the camera cannot be used, keyboard controls allow the games to remain playable.

Privacy by Design

Since camera access is central in our platform, privacy was an important consideration from the beginning. All computer-vision processing happens locally on the user’s device. Images and videos are never uploaded or sent to a server, as our application operates entirely without a backend server.

Challenges we ran into

Find a suitable Pre-trained Model

Finding a suitable pre-trained model that could recognise the gestures required by our games was challenging. Existing models may support different gesture sets or recognition approaches, so we need to find one that could be realistically integrated into our project within the 24-hour hackathon.

Gesture Recognition Accuracy

Recognition accuracy was another major challenge. Differences in hand position, orientation, movement, camera angle and individual signing style can affect detection. In addition, similar-looking gestures can also be hard for the model to distinguish reliably.

Real-time Latency of the detection

Since our mini-game relies on immediate interaction, gesture detection needs to respond quickly. Delays between performing a gesture and receiving feedback can disrupt the gameplay experience, particularly rhythm-based games. Therefore, we had to balance recognition accuracy with real-time responsiveness.

What we learned

During this hackathon, we learned how computer vision and real-time hand tracking can be incorporated into an interactive learning and fun experience. More importantly, we learned that accessibility does not always mean building technology that helps a minority adapt to the majority society.

We are happy and proud that we were able to turn this idea into a functional prototype that encourages people with no sign-language experience to take the first minor step towards communicating differently, and breaking down the communication barriers.

What's next for HandsUp

Mobile Experience

Extend the mini games to mobile devices, making the mini-games easier to access and allowing users to practice the sign language anywhere using their phone camera, promote to broader people.

Rhythmic game

Develop a rhythm-based game where players perform the correct sign in time with the beat, combining the sign-language learning with fast-paced gameplay

Multiplayer and Party Games

Expand the learning experience with more interactive multiplayer and party games, allowing groups of friends to learn, practice and have fun with the sign language together during parties or events.

Beyond our 24-hour prototype, we want to turn sign-language learning from something people feel they have to study into something they genuinely want to learn.