SharableWorlds

A 3D miniature studio in the browser, where you and an agent build on the same canvas.

What inspired me

I was tired of the usual AI 3D demo: you type a prompt, a pile of objects appears, and you’re left wondering if it actually understood what you asked for. There’s no real way to work with the scene. You just watch it happen.

WebMCP clicked for me because the page can be what the agent uses, not some API off to the side. Tools live on the site. A person can still be there, placing pieces, locking lots, arguing with the result.

I wanted that to feel what we were accustomed to. A grid. A bunch of assets. You say "a harbor," or you drop a boat yourself, and the agent has to work in that same language.

How I built it

Everything runs in the browser. Next.js as a static export, React, Three.js for the isometric view, Zustand for the world. It deploys on a Cloudflare Worker. No login. Nothing on a server that "is" the scene.

When the page loads it registers tools on document.modelContext. If something changes the world, it goes through one of those tools. Surprise Me button, share, the kit assets, and a connected agent all hit the same functions. Three.js only reads the state, the agent never talks to the renderer.

I made the agent go through and follow a sequence instead of dumping pieces:

get_occupancy
  → plan_scene
  → compose_scene
  → populate_zones
  → create_environment
  → inspect
  → validate_scene
  → repair_scene

It thinks in lots (C4), zones (picnic), and things like "east of the grill." validate_scene scores the result on six dimensions. Done means the score is high enough, not that a bunch of calls returned in an ok state.

The assets are Kenney, CC0. Roughly 1,800 pieces in 22 packs.

What I learned

An agent can call place_piece 47 times and still not build a picnic. I had been treating a successful tool call as progress. It is not. I ended up scoring the scene and handing back repairs, otherwise it would stop early and look finished.

Coordinates were a dead end. "Put this in the picnic zone, facing the grill" is something a model can actually use. (12.4, 0, -3.1) is not.

If the agent can knock over what you just placed, you are not collaborating. That is why locks on human placed pieces and "build on this" matter as much as the fancy compose step.

WebMCP being the only write path was annoying at first and then useful. I also needed to see what it was doing. The Details panel, the trace, the percentage all dove into what was happening behind the scenes. Otherwise it still feels like a black box, just a prettier one.

Challenges I faced

The first version technically used WebMCP and still sprayed pieces. My own runner was looping place_piece so the cursor had something to do. The model got a paragraph of instructions instead of a real sequence. Validation was so loose a picnic with no table could score 100.

So I split it up. plan_scene does not mutate anything. Layout and objects are compose_scene and populate_zones. Twenty rules, each with a { tool, args, why } if it fails. repair_scene runs those. build_scene is the whole loop for hosts that will not steer. extend_scene is there because "add a harbor" should not delete the village.

Then the small stuff started breaking the feeling of it. The cursor spawned under the kit and jumped when the camera moved. An empty board still looked like leftover scenery. You could not rotate the view. Share links replay a seed, which I wanted, until a leftover test URL made it look like a default world had kept rendering.

Most of those bugs were the same issue: the person and the agent are looking at one stage. If that stage is lying, the whole thing falls apart.

Getting registerTool working was the easy part. The work was making the tools a language two sides can share, and knowing when the harbor is actually a harbor.