SignLang Pacman

interpreted sentence translated to target language
signing sequences to form a sentence that is interpreted
Signing letters to form a word
Pacman maze starting

Inspiration

The Story of SignLang Pacman: Bridging the Silent Gap 🌟 The Spark: A Chance Encounter in Thailand The inspiration for SignLang Pacman didn't come from a classroom, but from a group tour to a local temple in Thailand. There, I met Greg, a fellow traveler. Throughout our journey - from navigating airport customs to ordering at bustling food stalls, I noticed Greg’s wife handled every heavy communication task. She later shared that because Greg had immigrated to several different countries, he never mastered a single sign language; every region has its own unique dialect and nuances. It got me thinking: What if she wasn't there? Greg’s independence was tethered to a human translator.

What it does

Pacman meets Sign Language. A gamified educational platform that teaches American Sign Language (ASL) and other sign languages through arcade mechanics and AI-powered real-time verification

How we built it

The application leverages Gemini 3’s advanced temporal analysis to decode complex sign sequences. By processing high-resolution frame bursts, the model performs linguistic decoding, mapping physical gestures to Gloss words and conceptual translations with high contextual nuance. In the Pacman-inspired game, Gemini 3 serves as a rigorous structural classifier. It evaluates hand landmarks against the ASL manual alphabet, ensuring learners maintain precise finger extensions and palm orientations. Furthermore, the project utilizes Gemini 2.5 Flash Image to dynamically generate instructional line-art diagrams. This feature provides immediate, personalized visual feedback for every sign encountered. These Gemini-powered capabilities transform the app from a simple tracker into a sophisticated linguistic engine, enabling a seamless transition between regional dialects while maintaining the educational rigor required for mastery.

The project uses a hybrid architecture: Game Engine: A custom engine was built using Next.js 14 and Zustand for state management. Players control Pacman by making specific ASL handshapes. "Linguistic Reasoning Engine": Gemini 2.5 Flash is used to analyze signs. The system performs "temporal analysis," processing 5-second videos to decode the movement and syntax of signs (Subject-Verb-Object). Visual Feedback: To provide learners with immediate feedback, Gemini 2.5 Flash Image is used to dynamically generate instructional line-art diagrams, providing a visual reference for every gesture.

Challenges we ran into

The primary challenge was the Sequence-to-Sequence (Seq2Seq) problem. Sign language involves handshape (h), location (l), and movement (m) over time (t): (S=f(h,l,m,t))Accuracy: Interpreting these sequences accurately was difficult. This was addressed by using MediaPipe for hand-tracking and offloading the "translation reasoning" to the Gemini 3 API. Rate Limits: Image generation is resource-intensive. To stay within API quotas, a "generate and store" strategy was implemented using Gemini. A 3-second cooldown was implemented on real-time requests.

What we learned

Sign language has aspects that go beyond the alphabet. The project demonstrates the use of initialization, where certain handshapes are incorporated into conversational motions. The technical development followed a progression: Phase 1: Building and refining LSTM (Long Short-Term Memory) models for sequence recognition. Phase 2: Experimenting with heuristics and API linking to improve accuracy. Phase 3: Replacing the manual training pipeline with the Gemini API. This removed the need to manage large datasets, allowing the model to handle the linguistic decoding.