Voxel WebMCP

An agent-native voxel sandbox: AI Agent builds, and explores your 3D world by calling WebMCP tools directly — no clicking, no DOM guessing.

Comment

Inspiration

The idea for Voxel Web MCP came from WorldEdit, a tool I used in voxel sandbox games as a kid. Back then, building anything large meant running around the world, selecting coordinates, and manually entering commands for every wall, region, replacement, or correction.

When we first saw WebMCP, we wondered what would happen if an agent could use similar geometric operations directly inside a browser-based 3D world. Instead of telling it which exact blocks to place, could we simply describe the final intention in natural language?

A voxel world felt like a particularly demanding test. It is not a form, a store, or a collection of buttons. It is a procedurally generated three-dimensional environment containing terrain, materials, structures, coordinates, and a moving player. If an agent could understand and meaningfully modify that through WebMCP, the same approach could potentially work across many other complex web applications.

What it does

Voxel Web MCP is a browser-based voxel sandbox where a human player and an AI agent can build together.

The player can make requests such as:

  • Build a seaside house with a pier.
  • Inspect this small architectural model and transform it into a full-scale structure.
  • Replace one material throughout a building.
  • Recreate an attached image as voxel pixel art.
  • Copy, move, rotate, mirror, or repeat an existing region.
  • Inspect the terrain around the player before deciding what to build.

The agent operates as a world editor around the player. It understands the player's position, viewing direction, targeted block, world boundaries, available materials, and selected regions of voxel geometry.

We deliberately did not create semantic tools such as build_house. The application only exposes reusable geometric primitives. The agent must decide for itself what should become a wall, roof, window, tower, road, or complete building.

This makes the experience open-ended: the tools are not tied to a collection of predefined demonstrations.

How we built it

We started with VoxelSrv, an open-source browser voxel game built on noa-engine and Babylon.js, and created our own WebMCP integration layer.

The application registers eleven structured tools through the browser's document.modelContext API. They cover:

  • operating guidance for the agent;
  • world boundaries and the material catalog;
  • player position and viewing direction;
  • compact region scanning;
  • batch block placement and replacement;
  • copying, moving, mirroring, rotating, and stacking regions;
  • WorldEdit-style undo.

The game runs both its client and authoritative single-player server inside the browser. The server runs in a Web Worker, while WebMCP calls travel from the main page to that worker through correlated messages. World edits are applied by the authoritative server and sent back to the renderer as normal block updates.

The world is stored locally in IndexedDB. Players can save, regenerate, export, and import worlds without requiring an external database or game server.

We also added a Docker build that compiles the Vite application in a Node.js stage and serves the final static bundle through Nginx.

The context problem

Writing blocks was not the hardest part. The real challenge was giving the agent an affordable and reliable understanding of a three-dimensional space.

The agent should not have to reconstruct the world from screenshots, and sending every voxel as raw JSON would consume an unreasonable amount of context. In one representative region, a raw dump required roughly 49,000 tokens.

We therefore designed a progressive inspection system. The agent can begin with a cheap summary, request a heightmap when it needs terrain information, or inspect selected two-dimensional slices when it needs to understand floors, walls, roofs, and elevations.

The application also provides an explicit coordinate basis: the player's forward, right, and up directions, along with the exact block being targeted. This allows the agent to interpret requests such as “behind me,” “in front of me,” or “continue to the right” without guessing camera geometry.

Challenges we faced

The upstream engine was built around an older JavaScript ecosystem. Making it work with a modern Vite build required pinning compatible versions of noa-engine, Babylon.js, the voxel protocol, and several legacy dependencies.

The game also uses multiple browser execution contexts. The renderer runs on the main thread, while the authoritative world and terrain generation run in Web Workers. WebMCP edits therefore had to update the server's real world state and remain synchronized with the client's chunk cache and rendered meshes.

Other challenges included:

  • keeping world inspection compact enough for an agent's context;
  • providing useful material descriptions without duplicating them across every tool;
  • making large edits visible without flooding the renderer;
  • supporting concurrent tool calls while preserving understandable undo behavior;
  • restoring reliable mouse input after browser focus changes;
  • adapting camera controls for agent browsers that cannot use relative mouse movement;
  • preserving worlds locally and adding portable import and export.

What we learned

The most surprising result was how much an agent could accomplish with a relatively small set of general-purpose tools.

It did not need a dedicated house generator or a library of predefined structures. Once it could inspect the terrain, understand the player, discover available materials, and perform geometric operations, it began producing coherent and often genuinely creative architecture.

We also learned that agent context is part of the product's interface. Tool descriptions alone are not enough for spatial work. Coordinate systems, compact representations, world limits, material semantics, and clear operating guidance all directly affect the quality of the result.

Most importantly, WebMCP allows the agent to stop being a chat window beside an application and become an active participant inside it.

What's next

The current version focuses on a single-player creative sandbox, but the same model can extend much further.

An agent could work with a normal player's inventory and permissions, assist multiple players in a shared world, automate level-design workflows, manipulate other kinds of 3D editors, or interact with entirely different WebMCP-enabled applications.

WebMCP creates a shared interface between agents and web applications. Compatible agents can change without requiring the application to be redesigned around each one, while the application can evolve without changing the basic way agents discover and use its capabilities.

Today, that application is a voxel world. Tomorrow, it could be almost anything on the web.

Built With

Share this project:

Updates

Submission history