Inspiration

We wanted to step inside a fairy tale instead of reading one. Most story games are fixed branches, and most "AI games" are a chatbot in a text box. We asked what it would feel like if an AI storyteller ran the game like a Dungeon Master: it writes the quest as you play, reacts to how you play, and hands you a storybook of your own adventure at the end. Then we wanted you to play it with your body and a wand instead of only a keyboard.

What it does

Storybound is a 3D third-person fairy-tale adventure. Two heroes share three hearts and earn stars. A Gemini-powered Dungeon Master narrates each chapter and sets challenges:

Freeze: hold perfectly still while the frost gazes.
Pose: strike the rune poses (arms up, arms out, crouch).
Vote: choose the route, and the story branches.
Combat and a boss: a three-hit sword combo, spells, a roll with invincibility frames, enemy camps that notice you and give chase, and a boss with a warning phase and a vulnerable window.

When the run ends, Gemini writes a personalized storybook from what you actually did: your choices, the challenges you passed or failed, and the hearts you lost.

How we built it

Story engine (Gemini): a FastAPI backend calls Gemini (gemini-3.8-flash) with structured output, so every story beat is validated JSON the game can run: narration, challenge type, options and outcomes. Thinking is disabled and requests have a 12 s timeout, which brought latency from about 5 to 11 s down to about 3 s.
Reliability: if Gemini is slow or unavailable, the backend returns a 503 and the game falls back to a scripted story, so a demo never freezes on a bad connection. The backend also stores telemetry and a leaderboard in SQLite with bounded, validated payloads.
3D game: Three.js with Rapier physics and a kinematic character controller. The camera is a spring-arm rig with five probe rays so it doesn't clip through walls, plus a dithered fade for scenery between you and the camera. Post-processing adds bloom, a storybook colour grade and vignette. Particles, procedural WebAudio (generative music, ambience, footsteps and combat sounds) and enemy AI (idle, alert, chase, return, respawning camps) are all built in-house.
Art direction: based on teammate Rishaan's storybook art: warm golden light, emerald greens and soft magic glow.
Computer vision: [teammate haicil's OpenCV tracker follows a yellow-tipped wand so spells can be cast by waving it. Describe exactly what works in your demo, or delete this bullet.]

Challenges we ran into

Making Gemini reliable as a game engine. Free-form text breaks a game. We moved to strict schemas, banned game-stat words in story prose, and built a mirror of the scripted story so Gemini only fills in text and cannot break the game flow.
A camera and movement that feel right. Early versions clipped through trees, swung around, and left the visible character running ahead of its physics body. We fixed it by measuring: we found the animations had forward motion baked in and removed it, and we tuned movement until direction changes were instant.
Stillness is hard. The physics engine jitters slightly, so a standing hero looked like it was moving and the freeze challenge could not be passed. We added a dead zone.
Security and safety basics. We bounded event payloads, restricted inputs to known values, kept the API key out of the repo, and left unlicensed third-party models out of the public repository.

Accomplishments that we're proud of

A story that is actually generated and validated live, with a safe fallback.
A 3D world with real combat, enemy camps, a boss and a roll/flash-step system, tested by automated gameplay checks (36 unit tests and 12 in-browser tests).
The storybook ending, which turns a run into something you can keep.

What we learned

Structured output makes an LLM usable as a game system, not just a narrator. Measuring behaviour (distances, timings, camera jitter) beat guessing at "feel". And a fallback path is as important as the AI path.

What's next for Storybound

Full camera-based body control and wand spells (mapping drawn shapes to spells).
Three players on their own laptops, and a phone as a motion wand.
More environments and a richer cast of Gemini-written characters.