NullPointers

Inspiration

To see depth in a 3D world you usually need a VR headset. To play with your body you need a console with a camera, like the $250 Nex Playground. Almost every laptop already has a webcam, so we tried to see how far a webcam alone can go.

While testing we noticed a second problem. When a gesture failed, players did not know why and gave up. Hand exercises at home have the same issue: in a systematic review, 30-50% of stroke survivors stopped their exercise programme within the first year (Levy et al., 2019). So the game tells you which finger is wrong.

What it does

Window is a 3D game in the browser. Your screen works like a real window: lean left and you see what is on the left, stand up and you look down into the level.

• Head: the picture is redrawn from your point of view.
• Right palm: a joystick that walks the hero. Right fist: jump.
• Left fist: you hold the world and rotate, tilt and zoom it.
• Two open palms: menu and pause.

Each level is built around one gesture. The lobby teaches the basics. "Islands" is walking and jumping. In "Peek" stars hide behind walls and you see them by raising your head. In "Turn the World" a bridge comes together from floating planks only at the right angle.

When a gesture is almost right, the wrong fingers light up red in the camera window and the voice says what to fix, for example "To jump, close your fist all the way. Not bent: ring, pinky". After each level the results screen lists your mistakes with a tip for each.

The game opens in English and has a Russian version, "Okno" (it means "window"), with a switch on the start screen.

How we built it

• Recognition runs in the browser with MediaPipe Tasks Vision: Gesture Recognizer (21 points per hand), Face Landmarker (478 points) and Pose Landmarker. It uses the GPU and falls back to the CPU. There is no server, and the video never leaves your device.
• We wrote our own gesture rules on the 3D hand points. Each finger is rated extended, half or curled by how far it bends, and fist and open palm are built from that. A One Euro filter removes jitter.
• MediaPipe often swaps left and right on a fist. We decide the side when a hand appears (body pose, memory, position relative to the face) and then follow the hand by its continuous motion. A new hand counts only after 3 frames in a row, which removes ghost hands on the face.
• For the window effect, the head position relative to the screen sets an off-axis projection, so the picture edges stay on the screen edges like a window frame. The idea is old: Johnny Lee showed it with a Wii remote and infrared glasses in 2007. We do it with a webcam only and use it as a game mechanic.
• The game uses React, Three.js (React Three Fiber), Rapier physics, TypeScript and Vite. The world was modelled and rendered in Blender, and textures come from Poly Haven (CC0).
• Voice: 45 lines in two languages. The Russian voice was made with ElevenLabs. The English voice was generated on our laptop with the open model Kokoro-82M, and a Whisper check confirms all 45 files match their text.
• 155 automated tests (Vitest). About 13 ms per camera frame and 120+ fps on our laptop.

Challenges we ran into

• Telling the left hand from the right one. This was the hardest bug.
• The first joystick sat next to the face. Our arms got tired and the circle got lost in the camera view. We moved it low and to the right.
• The tracker processed the same camera frame 2-3 times. After the fix, frame processing became twice as fast and rotating the world became smooth.

Built during the event

Our first commit is from 28 September 2026 (commit e0d1073). EurekaDev submissions opened on 27 September. All code in the repository was written after that. For EurekaDev we added the English interface and the English voice.

How we used AI

We wrote most of the code with Claude Code, an AI coding assistant, following our own plan. The idea, the game mechanics, the level design, the gesture rules and thresholds, the Blender world and every decision were ours, and we checked the game by hand on a live camera.

What's next

The game already measures how far each finger bends in every frame. Studies suggest this is a usable signal: MediaPipe finger angles were repeatable across repeated measurements of the same videos (ICC above 0.75, 48 healthy hands, Létourneau et al., 2024) and within 5.14 degrees of a goniometer on average (Gu et al., 2026). A 2025 Cochrane review found that video games added to usual rehabilitation probably improve arm function (190 trials, Laver et al.).

Next step: a per-finger report after each session and a pilot with a physiotherapist.

Limitations

• Chrome or Edge, a webcam and light on your face.
• The window effect is correct for one viewer.
• Window is not a medical device and has not been tested with patients.

Built With

• blender
• elevenlabs
• kokoro
• mediapipe
• rapier
• react
• react-three-fiber
• three.js
• typescript
• vercel
• vite
• vitest
• webassembly