Math Attack

Inspiration

Math practice usually looks the same: a worksheet, a timer, and a lot of sitting still. We wanted to make arithmetic feel like an arcade game instead, something you play with your whole body rather than solve on paper. We also know that the recent generations are lacking in fine motor dexterity in which the controls for this game will help develop them. Webcams are in almost every laptop, and in-browser hand tracking has become fast enough to follow every finger in real time. That made us ask: what if your hands were the controller? Number Rush grew from that idea. You grab numbered cannonballs out of the air, smash them together to add, subtract or multiply, and fire the result at incoming pirate ships before they reach the shore. We wanted a game where getting faster at mental math feels like getting better at a game, not like studying. And because all it needs is a browser and a camera, anyone can jump in without downloading anything or buying special hardware.

What it does

Number Rush is a webcam-controlled math arcade game that runs entirely in the browser, with no controller, download or special hardware. Before each game you pick a mode: Addition, Add & Multiply, or Challenge, which mixes addition, subtraction and multiplication and gets harder every 60 seconds. Next you line your hands up with on-screen outlines so the camera can track them. Then pirate ships sail toward the shore, each carrying a number. Numbered cannonballs roll along a rack at the bottom of the screen. You pinch one to pick it up and touch two together to combine them with the active operation. Lifting your thumb switches between operations. When you've built the right number, you drop the ball into the cannon facing that ship and fire it to sink the ship. A wrong number just bounces off. Birds also swoop in to block your view, so you open your other hand to pull out a scythe and chase them off. You have three hearts, lost when a ship reaches the shore or a bird hits you. Every ship you sink scores its number in points, and every bird scores 10. Players can create an account and compete on a separate leaderboard for each mode.

How we built it

Number Rush is a React + TypeScript app built with Vite. The game world is rendered in 3D with Three.js through React Three Fiber and drei, layered on top of the live webcam feed so it feels like simulated AR. The ships, cannons, rack, birds, gloves, scythe and beach are custom models built in Blender and exported as glTF. Hand tracking uses Google's MediaPipe Hand Landmarker running entirely in the browser, so no video ever leaves the player's computer. Every frame we read 21 landmarks per hand and turn them into gestures with our own rules:

pinching the index and middle fingertips grabs a ball
lifting the thumb off the curled ring and little fingers switches the operation
fully opening a hand summons the scythe Each gesture has to be held for a moment before it counts, which smooths out racking jitter and stops accidental grabs and drops. A calibration screen checks each hand's position and size against an on-screen outline, so players start at the right distance from the camera. Game state lives in small shared stores that are updated every frame and read by both the 3D scene and the React interface. That kept the arithmetic, collisions, ships, birds, lives and score in sync without slowing down rendering. ## Challenges we ran into Turning gestures into reliable controls. Hand tracking is noisy, and a gesture that works in a demo often fails in real play. Our first operation switch fired when the thumb looked far from the palm. Players kept switching by accident whenever they turned a hand sideways, because from the camera's angle the thumb looked stretched out. We redesigned it around the resting pose: the thumb rests on the curled ring and little fingers, and you switch by lifting it off. That distance stays small however the hand is turned. We also made every gesture hold for a moment before it counts, so flicker and tracking dropouts can't grab, drop or switch anything. Calibration that didn't recognise aligned hands. Our hand-alignment screen kept saying "Line it up" even when a player's hand sat right inside the outline. The outline's wrist was drawn in a different spot from the wrist point MediaPipe actually tracks. On top of that, MediaPipe sometimes labels left and right hands the wrong way round. We fixed the outline geometry, assigned hands by where they are on screen instead of trusting the left/right labels, and added on-screen dots and target rings so players can see exactly what the camera sees. Lining up the camera, the 3D scene and the hands. The webcam feed is mirrored and cropped to fill the screen, so camera coordinates, screen pixels and the 3D world all disagree. Getting a ball to sit exactly between your fingertips meant carefully converting between all three. Visual glitches between modes. Gloves kept appearing over the scythe because the gesture classifier flickered between poses for a few frames. We had to decide carefully which gesture "wins" each frame. ## Accomplishments that we're proud of It actually feels like a game. Grabbing a cannonball out of the air, smashing two together to make a number, and watching a ship sink is genuinely fun. Players forget they're doing mental math. That was the whole goal, and we got there. Screen time that gets kids moving. A 2025 systematic review of 24 studies found that excessive screen time in early childhood is mostly linked to weaker motor development, but that the effect "varies with screen content and environment" (Bakht et al., Pediatric Discovery, 2025). Number Rush can't be played sitting still. Pinching fingertips to grab, lifting a thumb on its own, and coordinating both hands at once turn math practice into fine motor practice too. No controller, no download. It runs entirely in the browser with just a webcam. All hand tracking happens on the player's own device, so no video is ever sent anywhere. Gesture controls that hold up in real play. We built our own gesture rules on top of MediaPipe's raw hand landmarks and kept refining them until they stopped misfiring when hands turn, jitter or briefly drop out of view. A full stack that works end to end. Our own Express API and Supabase database handle accounts and leaderboards, with hashed passwords and secrets kept on the server. If the server is down, the game keeps working offline. ## What we learned Hand tracking is the easy part; gestures are the hard part. MediaPipe gives you 21 points per hand almost for free. Turning those points into controls that feel intentional took most of our effort. We learned to judge gestures by comparing points on the hand to each other, rather than by how the hand looks to the camera. A check like "is the thumb far from the palm?" breaks as soon as a hand turns. A check like "has the thumb left the fingers it was resting on?" doesn't. We also learned that every gesture needs a short hold before it counts, plus a gap between the "on" and "off" thresholds, or noise in the tracking turns into accidental grabs and switches. Design the controls around a resting pose. Our biggest breakthrough was defining a comfortable "home" position for the hands and making every action a small, deliberate move away from it. That cut down on accidental input and made the game much easier to teach. Coordinate systems will get you. A mirrored, cropped webcam feed, screen pixels and a 3D scene each measure position differently. We learned to pick one shared coordinate system and convert everything into it in a single place. Small config details can break a whole feature. Our leaderboard went down because of one missing https://. Testing the deployed app, not just the local one, matters. ## What's next for Math Attack Division and new challenges. Add division with whole-number answers, plus timed rounds and bonus targets that reward solving with fewer combinations. Throwing with your hands. Right now you drop a ball into a cannon. Next we want players to throw it, using the speed and direction of their hand movement to aim. Adaptive difficulty. Track which number facts each player finds hard and spawn more ships that practise them. That turns the game into personalised practice, not just a fixed difficulty ramp. Classroom mode. Let teachers create a class, choose which operations and number ranges students practise, and see progress over time. A class leaderboard would turn practice into a friendly competition. Two-player mode. Two players defending the shore side by side on one camera, either working together or racing each other. More ways to play. A mouse and touch fallback for devices without a good camera, accessibility options such as one-handed controls and adjustable gesture sensitivity, and sound effects and music for more arcade feel. Measuring the impact. We'd like to work with educators to test whether Number Rush improves arithmetic fluency, and whether its hand-based controls support fine motor skills compared with traditional screen-based learning apps.