Morphus - the 3D world editor your agent can operate

A WebGPU 3D world editor that hands its real authoring tools to your browser agent. Sculpt terrain, grow forests and build scenes by asking...the scene graph is the API, not the pixels.

Comment

Inspiration

Most attempts to let an AI agent drive a creative tool try to make the model see the screen: screenshots, pixel coordinates, guessing where a button is and hoping the click lands. For a 3D editor that's close to hopeless, because the interesting state isn't on screen at all. It's a scene graph, a modifier stack, a spline running through a forest field.

Meanwhile our editor already had a complete, precise description of everything it could do. Tool declarations written for its own built-in Copilot, each with a JSON Schema. That surface existed, and it was locked inside one chat panel.

WebMCP is the piece that was missing. It lets a page hand its real command surface to whatever agent the user is already talking to. So the question stopped being "how does a model learn to use our UI" and became "why is our UI the interface at all?"

What it does

Morphus is a WebGPU 3D world editor that runs entirely in the browser. It sculpts terrain, grows forests, blocks out levels, casts combat VFX, and lays out procedural cities.

With WebMCP it registers all 154 of its tools with the browser, so an agent can build a world in the page the person is looking at. No server, no API key, no separate MCP process.

Ask it for a landscape and it calls create_mesh_terrain, then terrain_sculpt_stroke along a path of world-space points, then create_forest_field and grow_forest_field to plant a stand of trees along a spline.

Ask it for a city and it runs a whole pipeline. generate_street_grid lays avenues and cross streets with real junctions, kerb returns and lane markings. generate_city_massing divides the blocks between them into lots that front the street and extrudes them into volumes. generate_city_buildings replaces the tallest of those with actual architecture: limestone banks, setback towers, corner headquarters with colonnades and cornices.

Throughout, it reads back with get_terrain_state, get_street_network and list_nodes, so it acts on what's really in the scene rather than on what it assumed.

The scene graph is the API. A heavy GPU application becomes operable by a model that never has to render a frame.

How we built it

The core decision was to build a bridge, not a second implementation:

await document.modelContext.registerTool(
  {
    name: "terrain_sculpt_stroke",
    description: "Sculpts the terrain along a path of world-space points...",
    inputSchema: { /* JSON Schema, shared with the in-app Copilot */ },
    annotations: { readOnlyHint: false, untrustedContentHint: true },
    execute: async (input, { signal }) => { /* ... */ }
  },
  { signal: controller.signal }
);

WebMCP asks for exactly the shape the Copilot declarations already had, so src/lib/webmcp/tools.ts reuses both the declarations and the same executor. A browser agent and the in-app Copilot run identical code paths and cannot drift apart.

The payoff showed up when we built the city. Nine new tools became agent-drivable the moment they existed, with no bridge changes, no registration code, and no separate schema to maintain. The surface went from 145 to 154 by writing the features.

Everything else followed the platform's guidance rather than working around it:

  • Annotations derived, not hand-listed. readOnlyHint comes from the tool's name prefix, because a hand-kept list of read tools beside 154 declarations goes stale the first time someone adds a get_. One tool lies about itself: capture_mesh_modeling_base captures topology as the base of a live modelling stack, so it writes. It's named as an explicit exception. A tool wrongly marked read-only is worse than one left unmarked, because it invites an agent to call a mutation speculatively.
  • Lifetimes via AbortSignal. There is no unregisterTool. Aborting the registration controller is how tools go away, so they disappear cleanly with the editor. A second signal on execute cancels a call mid-flight, which matters when a sculpt covers a 4 km terrain.
  • A visible record. Tools run in the page, so the only trace of a call is the one the page chooses to show. The menu bar carries an AGENT readout: tools registered, and the name of the last call.

Challenges we ran into

We shipped a curated 20, then reversed it. The original argument was sound. A tool list is a prompt, and every entry spends the agent's attention. But it cuts the other way at this scale. The editor's real capability is the whole surface, and an agent that cannot inset a face or lay a street is operating a demo of the editor rather than the editor. Curation became the agent's job, and ours became making that job possible: order the list so read and world-building tools lead, and annotate every entry honestly.

Character budgets are a design problem, not a formatting one. Chrome recommends 500 characters per description. Truncating the overruns turned out to be exactly wrong, because the tail of a good description is the part that says when not to reach for the tool. We rewrote those by hand, aliased four over-long names, and wrote a budget check that fails the build. The same logic applied to results. A silently truncated result reads as a complete answer, which is how an agent ends up confidently acting on half a scene graph.

A forest was planted thirty metres underground, and reported success. The viewport draws two different terrains depending on renderer backend, and they are not the same surface. The forest sampled one; the viewport drew the other. Measured, they differ by 26 to 30 metres, so every stand was buried and every tool call returned "success": true. There is now exactly one function that answers "how high is the ground here", and everything placed on the ground goes through it, so forests, streets and buildings can only ever be wrong together.

Roads that bridge their own kerbs. Sampling terrain once at a street's centreline and reusing that height across its full width buries the uphill kerb and floats the downhill one by half the width times the gradient. On a 16m avenue that was 2.3 metres of float. Sampling per edge took the worst deviation across 228 vertices from 2.303m to 0.000m. We later replaced the hand-rolled ribbon entirely with three-roads, an OpenDRIVE-style model that owns lanes, kerb returns and paint properly.

A building grammar that can't build a city. We vendored procedural-bank, a shape grammar producing early-twentieth-century commercial architecture, and measured before trusting it: 142,552 triangles from 478 kit modules per building, in about 300ms. Turning every knob down, with no ornament, no colonnade and no crown, reached 117,660. A saving of 17%, because the cost is in the modules and not the decoration. Building all 456 volumes of a modest grid would be roughly 62 million triangles and two minutes of blocking work. So it generates a few dozen landmarks standing among massing boxes, which is what a downtown looks like anyway.

Resizing a canvas is not resizing a renderer. On WebGPU the render targets and depth texture are allocated when the renderer is built, so a canvas that grows afterwards leaves them behind and every frame fails validation against a black viewport. The sting is in the fix. The poll that waits for a real size cannot use requestAnimationFrame, because a page that isn't being composited doesn't run animation frames at all. The one mechanism that looks right is guaranteed to be asleep exactly when it's needed.

Testing an API most browsers don't have. WebMCP needs Chrome's origin trial or chrome://flags/#enable-webmcp-testing. So we built a stub. ?webmcp=stub installs a minimal modelContext plus a console harness, exercising the same registration path and the same tool code with no model in it.

window.__webmcp.list().length;                    // 154
window.__webmcp.describe("generate_street_grid"); // the schema an agent sees
await window.__webmcp.call("generate_street_grid", { columns: 4, rows: 3 });

It settles the deterministic half of the question, "did this tool run and return what it promised", that Chrome's own guidance says to answer before writing evals.

Accomplishments that we're proud of

  • No parallel implementation. The agent's tools and the app's tools are the same tools. Nine city tools shipped with zero bridge changes.
  • Budgets enforced by a check that fails the build, rather than by good intentions.
  • The human stays the one who understands the document. Geometry appearing in a 3D scene is ambiguous. Did the person do that, or the model? The activity readout answers it on every call.
  • It degrades cleanly. In a browser without WebMCP the editor behaves exactly as before: no crash, no chip, no console noise.
  • Bugs closed by measurement, not by eye. Planted heights matching terrain to 0.000m, road vertices conforming to 0.000m, triangle budgets sampled across seven configurations. On a project where the camera can start below the terrain surface, "I can't see anything" is not evidence.

What we learned

The lesson we expected was "curation is the feature". We shipped that, then reversed it, and the reversal is the more useful finding. Curation is right when the catalogue is the product. When the catalogue is a capability surface, withholding tools just makes the agent worse at a job it was going to attempt anyway.

The second was that WebMCP's constraints are mostly good design advice with teeth. The character budgets forced descriptions that say when not to use a tool. The annotations forced us to be explicit about which operations are safe to call speculatively.

The third had nothing to do with agents. A tool that returns success: true is not evidence that anything happened. A buried forest, a white tree, a blank viewport and a building whose vertices were all NaN each reported success. Every one was found by measuring the thing itself.

What's next for Morphus - the 3D world editor your agent can operate

  • A residential vocabulary. The building grammar makes banks and towers. It has no brownstones, tenements or warehouses, so the periphery of a generated city is still massing boxes. That gap is the clearest next win.
  • Traffic. The road network already carries lanes and junctions, which is most of what agents need to move along it.
  • Evals. Deterministic tests only cover half of this. Next is a dataset of direct prompts ("make a ridge running east to west") and open-ended ones ("give me somewhere an ambush would work"), scored on tool choice, arguments, and whether multi-step journeys survive a mid-chain failure.
  • exposedTo for cross-origin collaboration, letting a companion tool drive the editor without the user leaving it.

Built With

Share this project:

Updates