StoryScape

Inspiration

Have you ever drawn your dream world as a kid? How about being able to step into it? We wanted to build the special thing little Harold drew the other day. Instead of just a picture, an actual 3D world you can fly through, with the castle he placed, and the sky the colour he picked.

What it does

Drawing

draw using a whiteboard. you get a variety of colours, brush sizes, an eraser, fill bucket, shapes, and under/redo; works with mouse, touch, or stylus

Interpreting

a vision model reads the messy drawing, turning it into a world brief

Build

agents build the world by calling tools that edit a JSON scene
full scene is streamed to the browser allowing you to see live edits
mix of library models + generated models

Explore

a low-poly world with unique lighting, terrain, weather, scenery, structures
auto fly through visits at each point of interest, or control the camera yourself

How we built it

The AI only decides what's in the world. It never draws anything or picks a colour. The renderer does all the visuals, and a zod schema is the only thing the two halves share.
The drawing board is a canvas with perfect-freehand for strokes and a wax grain pass on top so it looks like crayon instead of a vector line. Pen, eraser, fill, shapes, undo. Done exports a PNG.
The PNG gets moderated, then a vision model lists what's on the page: what each thing is, where it sits, how big, what colour, and how sure it is. We tell it to take the drawing at face value, so a six-legged purple thing stays a six-legged purple thing.
A second call turns that list into a brief: zones, the important objects, mood, weather, time of day, and a three-colour palette taken from the drawing itself. Each object gets matched to the closest model in our low-poly library.
Then an agent builds the world with tools (create_zone, place_object, scatter, that kind of thing). Each call edits a JSON scene that gets validated before it goes anywhere. If the agent makes a bad call the error comes back to it and it tries again. Page position maps straight onto the ground, left is left, bottom is near, top is far, so the world comes out laid out the way the kid drew it.
The full scene streams to the browser over SSE after every tool call, which is how you get to watch it build. A skybox generates in the background and swaps in when it's done.
The renderer is React Three Fiber. Terrain is seeded noise that drops into sea at the edges so every world is an island. One sun, angle and colour from the time of day. Coloured fog, bloom, ACES tone mapping, actual falling snow and rain. Models get normalised on load (scaled to a target height, origin at the feet) so the agent never has to know how big a .glb is. Everything is keyed by id so a new scene only mounts what changed.
Every number you'd want to tune lives in one presets file. There are no magic numbers in the components.
TypeScript, Next.js, React Three Fiber, drei, postprocessing, Hono, the OpenAI SDK, zod.

Challenges we ran into

We originally wanted a whole agent workflow to built the 3D world on its own, but realized quickly that it would take too long to generate the assets and the cost would build up. So we flipped the problem: instead of generating geometry, the agent composes worlds from a curated low-poly library (CC0 packs plus a few models we built ourselves and baked to .glb), and the one thing we do generate is the sky, as a 360° image, since image generation is cheap and reliable.

Accomplishments that we're proud of

Our original idea with this project was to input any story instead of drawings, but as we built out some of the foundations and discussed the idea more and more, we realized that we could generalize our idea for any world, and instead of intepreting English, we could interpret a drawing. Pivoting halfway through a hackathon was scary, but the foundation held: the renderer still worked the same with the input changing, and by the end a kid's scribble of a castle on a cloud came out as a castle on a cloud.

What we learned

Give the AI choices, not numbers. The agent picks dusk or desert by name and the renderer decides what that looks like. When we let it pick raw values (colours, sizes) the worlds came out plausible and slightly wrong every time. Enums plus a few well-chosen numeric fields worked far better than freedom.
Mood comes from lighting, not models. Ten low-poly models with the right palette, fog and bloom read as a world. Sixty mismatched models with flat lighting read as a pile of assets. We spent our time on the former.