Craftpilot

Inspiration

Someone put Git inside Minecraft and we couldn't stop thinking about what else you could cram into the game. Big builds take forever by hand, so we wanted a copilot: you say what you want, it shows up in front of you.

What it does

Type /build a cozy cottage with a stone chimney (or a skyscraper, or a dragon statue) and CraftPilot builds it in your world, layer by layer, right where you're standing. Don't like it? Say "make it three floors" or "make the roof red" and it fixes it. You can also just save it as a Litematica schematic.

How we built it

Two different agents, because buildings and statues are totally different problems. For buildings, an LLM writes a structured "build program" (shape, roof, windows, palette) and a Python engine turns that into actual blocks using real Minecraft builder tricks like gradients, depth and texturing. For statues and organic stuff, we generate an image from the prompt first and have an LLM turn that into a voxel model. A small Fabric mod connects the game to the Python service and places the blocks one layer per tick. We built basically all of it with Claude Code.

Challenges we ran into

Making the builds not look bad. Early versions had floating blocks, windows on corners and rooms you couldn't walk into. We ended up writing a lot of very specific rules to fix that: flood-filling rooms to make sure they're reachable, only putting windows where they'd actually be visible, checking palettes against a 60-30-10 colour rule, filling roofs so the corners don't have gaps, and checking every block against what the game will actually accept.

Accomplishments that we're proud of

The two-agent setup really works. The text → image → LLM pipeline handles the weird organic builds, the procedural generation + LLM agent handles anything architectural, and between them we can cover a lot of prompts. And it's all live in-game with one chat command, which still feels a bit like magic.

What we learned

LLMs are great at deciding what a building should look like and terrible at placing thousands of blocks consistently. Everything got better once we let the model make the high-level design decisions and handed the rest to deterministic code. Also, we learned way more about Minecraft build theory than we expected: palette, depth and silhouette matter far more than size.

What's next for CraftPilot

A feedback loop where the model looks at a rendered preview of its own build and fixes what's wrong. Better facade detail (we were eyeing wave function collapse). And going bigger than one building: terrain, roads, whole villages. Undo would be nice too.