Inspiration
We were inspired by Google’s Magic Cat Academy Doodle and its simple, satisfying mechanic of drawing symbols to defeat enemies. We wanted to bring that magic off the screen, letting players cast spells with their own hands using only a webcam. We also wanted to address the boredom that can come with repetitive practice when learning a silent, visual language like ASL. SpellCam turns fingerspelling practice into an active challenge: every handshape becomes a spell, and every approaching monster gives players a reason to keep learning. That idea grew to include math and written characters, all within a playful paper-and-pencil world
What it does
SpellCam is a browser-based survival game where you play as a stick figure defending a sheet of paper from approaching paper monsters. Pinch your thumb and index finger to draw in the air, then release to cast. Each monster carries a sequence of symbols, and matching its next symbol removes it. One well-timed cast can hit several monsters at once. Players can choose from six subjects: Shapes, ASL fingerspelling, Math, Korean, Japanese, and Chinese. Survive increasingly difficult waves, face bosses, and use gesture-activated spells to turn the battle around. In Duel mode, two players compete directly, sending extra monsters to each other through successful attacks.
How we built it
We built the game with TypeScript, Phaser 3, and Vite, with all gameplay and recognition running in the browser. MediaPipe Tasks Vision tracks hand landmarks through the webcam. We turn those landmarks into drawing input using a custom pinch detector, a One Euro smoothing filter, and a direction-aware adaptation of the $1 Unistroke Recognizer. Preserving orientation lets the game distinguish a horizontal line from a vertical one, or a V from an upside-down V. For ASL fingerspelling, we trained a small neural network using Python, scikit-learn, and MediaPipe landmark data. We exported its weights to JSON and implemented inference in TypeScript. Players can also record personal examples to adapt recognition to their hands. Multiplayer uses PeerJS and WebRTC for direct browser-to-browser communication. Procedural canvas artwork and Web Audio sound effects support the paper-and-pencil style.
Challenges we ran into
The biggest challenge was turning noisy webcam observations into reliable, responsive controls. Pinching and releasing can introduce unwanted movement into a stroke, while tracking gaps can split one drawing into several. We built an input lab with calibration, recording, replay, and diagnostics to inspect these problems and tune the controls. Recognition also required balancing forgiveness with precision. Players should not need to draw perfect shapes, but different symbols still need to remain distinct. ASL introduced another challenge: strong performance on familiar training data did not guarantee the same results for new hands and camera conditions. Independent evaluation helped reveal weaker letters and motivated personalized examples and a recorded resting-hand pose. For multiplayer, we also had to coordinate readiness, shared settings, disconnects, and rematches across two browsers.
Accomplishments that we're proud of
- Making webcam-controlled spellcasting work entirely in the browser.
- Connecting one core game mechanic to six different subjects.
- Training and shipping a compact ASL classifier without a browser ML framework.
- Building direct multiplayer duels with shared settings and gesture-based spells.
- Creating a cohesive paper-and-pencil world with procedural artwork and sound effects.
- Adding accessibility options for reduced motion, color-blind palettes, larger cards, and more readable text.
What we learned
We learned that gesture interaction depends as much on timing and feedback as it does on recognition accuracy. A correct prediction can still feel frustrating if casting is delayed or the game does not clearly show what it understood. We also learned to evaluate machine learning beyond a single accuracy score. Testing on independent data exposed limitations that a held-out training split did not fully capture. Finally, building calibration and replay tools showed us how valuable it is to make an input system observable. Seeing exactly when a pinch starts, a stroke ends, or a prediction changes made iteration much more focused.
What's next for SpellCam
Our next priority is testing with more players, webcams, and lighting conditions to improve gesture reliability and ASL recognition. We want to strengthen difficult letters, explore motion-based recognition for J and Z, and improve the reference cards. We also want to expand the lessons, check character placement alongside stroke order, and refine difficulty progression. For multiplayer, future work includes more reliable connectivity, stronger competitive validation, and a complete ranked experience.