VibeBuild

Imagine vibe coding, but for Minecraft builds. VibeBuild is your agentic AI build crew, turning any text or image into a detailed structure in minutes, then dropping it straight into your world.

Comment 3

-- please click through all the images to see what vibe code has made so far!! --

Inspiration

We fell in love with the new agentic developer tools (think “vibe coding” with Claude Code): you describe intent, and a system plans, executes, and iterates with you. At another hackathon we saw a winning project that brought Git into Minecraft, and it sparked the same thought: Minecraft is already the ultimate creative canvas, similar to coding, so why not bring agentic workflows into building? VibeBuild is us mashing together the best of both worlds into something everybody wish already existed.

What it can do

Wanna recreate your bass guitar in Minecraft? We've already done that, simply upload a picture. How about Elmo? Or maybe a modern house, or a steampunk factory? Yeah, we've already created that in the fraction of the time it would take a human, with a higher quality result.

The Agentic Workflow

  • VibeBuild turns a text prompt or reference image into a fully-built Minecraft structure in minutes.
    • The player decides to create a prompt or upload a reference image, and the agents instantly begin.
    • The player reviews the result in a dedicated build dimension, then drops it into their world exactly where they want it using ghost preview placement (rotate, adjust height, place).

How we built it

  • Agentic pipeline: Image Agent → Planner Agent → Executor Agent → Finalizer Agent, each stage stays specialized and reliable. (more info about the agents on our github readme)
  • Models: Claude Opus 4.6 for supreme reasoning in planning/execution; Claude Sonnet 4.5 for the image agent (image + notes → build prompt). These models had the best results, but also compatible with other Anthropic, OpenAI, and Gemini models.
  • Structured tool calls: we use Zod schemas to define tools and enforce clean JSON input/output, so the model can issue strictly verified typed commands to Minecraft.
  • Realtime loop: a Bun backend streams everything over WebSockets (steps, tool calls, tool results) to keep builds interactive and live.
  • In-game execution: a Fabric mod receives tool calls and runs them via WorldEdit and custom commands, builds in a safe build dimension, then lets players ghost-preview, suggest any changes, and place the result back into their world.

Challenges we ran into

  • Keeping quality consistent while shrinking context and avoiding compaction like modern tools use to reduce tokens without degrading build coherence.
  • Teaching LLMs to not only create a good design, but also break it into plans, and have the spatial math and reasoning (our solution is called SPP) to properly execute these plans (plain text -> 3d world)

Accomplishment that we're proud of

  • Our agentic pipeline one shots extremely difficult builds, and creates a workflow extremely similar to popular vibe coding tools.

What we learned

  • Agentic tool schemas are expensive: every unused tool is a tax paid on every executor call.

What's next for VibeBuild

We want to change the way everyone builds in Minecraft. Vibe building is for everyone!

Tracks we're going for:

Magical Hack (cmon, this is hella magical), AI/ML immersion hack (no need to even imagine this... it's right in front of your eyes!!), and creative track because, yeah, this is pretty creative (especially considering it works in creative mode ;) )

Built With

Share this project:

Updates