Inspiration
Everyone has made a shadow puppet on a wall. It is the oldest hands-only game there is, and it needs no instructions. I wanted a VR game where hands are not a replacement for controllers but the whole idea: you make a shape, and something comes alive.
What it does
Shadowplay is a seated, hands-first puzzle game and sandbox. A lamp glows in front of you and casts the shadow of your real, tracked hands onto a paper wall. You make a bird, a rabbit or a wolf with your fingers, and the shadow story responds.
- Story mode: three short wordless scenes. A bird wakes a sleeping child, a wolf scares a crow off the fence, and a rabbit hops home to its burrow. Each scene takes about 30 seconds, so a full run fits inside one short break.
- Shadow Theater: a free-play mode. Each pose you hold calls a living animal onto the wall where your shadow falls. Birds flap and fly, rabbits hop, wolves prowl. The animals react to each other: the wolf chases rabbits, and birds and rabbits flee from it, so no two shows are the same.
- Pinch controls: pinch to start the game, and pinch the lamp to drag it, which changes how your shadow falls.
How I built it
Shadowplay runs on the Immersive Web SDK (WebXR) with TypeScript and is hosted free on Netlify, so judges open one link and play.
- Pose recognition reads the 25 hand joints WebXR provides. It measures how extended each finger is and requires a pose to be held for half a second, so there is no machine learning, no model download and no latency.
- The shadow is real geometry. Every joint is projected from the lamp's position onto the wall plane, so moving the lamp or your hand changes the shadow naturally.
- All art and sound are original. The paper-cut scenes are drawn procedurally on a canvas, and the sound effects are generated with Web Audio. There are no third-party assets and no external services, so it passes the "take it away" test.
Challenges
- The XR simulator exposes controllers rather than finger poses, so I added keyboard shortcuts (1, 2, 3) that force a pose. This let me test the whole game loop, and it also lets judges without a headset play from a desktop browser.
- The starter template loaded a demo robot from a CDN that timed out. I removed all starter content so the game has no runtime dependency on any external asset.
- Pose rules had to be forgiving enough for different hand sizes, so every threshold lives in one config block that is easy to tune.
What I learned
Hand tracking feels best when the hand shape itself is the interaction. Projecting real joints into a shadow made a simple pose feel magical, and short sessions with an instant payoff suit seated VR.
What's next
Mixed-reality passthrough so the lamp lights your real room, more animals and poses, shareable player-made puzzles, a daily challenge and a shared two-player wall.
Built With
- hand
- immersive-web-sdk
- meta-quest
- netlify
- three.js
- typescript
- vite
- web-audio-api
- webxr