AI Minecraft Builder

bulldog
dinosaur
bulldog with G (thumbnail)
house

Inspiration

As Computer Science majors, we seem to have two distinct personalities: the gamer who loses hours building in Minecraft, and the engineer obsessed with the latest breakthroughs in Generative AI. We asked ourselves: Why keep these worlds separate?

Instead of spending hours placing blocks one by one, what if you could just cast a "spell" with words and watch the structure appear? That was the spark for our AI Minecraft Builder.

What it does

Our project turns text into reality within the Minecraft universe.

User POV: A player enters our Minecraft server, and by casting /createbuild, the user receives a magical wand. Click the ground, type a prompt like "a giant bulldog with a top hat," and select your desired scale. Within moments, the AI-generated model is voxelized and constructed right before your eyes.

The Magic: The system interprets this prompt, processes it through our AI pipeline, and magically constructs a fully realized, voxelized 3D model of that object directly into the game world next to the player.

How we built it

We built a multi-stage pipeline leveraging the power of AWS. Here is the flow from text to blocks:

Text-to-Image: When the server receives the user-inputted prompt, it sends the prompt to an AWS backend. We use the Amazon Titan Image Generator to create a high-quality isometric view of the requested object. We automatically remove the background from the generated image to ensure a clean model.
Image-to-3D: We feed the clean image into HunYuan3D 2.1, a model that converts the 2D reference into a 3D mesh with texture.
Voxelization: We then take the 3D mesh and "voxelize" it—translating smooth geometry into Minecraft block coordinates.
Rendering: Finally, the coordinates are sent back to the server, and the blocks are placed in the world.

Challenges we ran into

Dependency Hell: We initially built our prototype locally using many specific Python modules. When we tried to move to AWS Lambda, we realized the environment didn't support our heavy dependencies. This forced us to scrap significant progress and rethink our plan.

Latency & Timeouts: Generating 3D models is computationally expensive. The time it took for the programs to complete was initially too long for us to work with.

File Size Management: We attempted to resolve our dependency issues by bundling a virtual environment, but the resulting package exceeded AWS Lambda's deployment size limits.

Token Management: We had to carefully monitor our API token usage to keep the project viable during testing.

Accomplishments that we're proud of

Taming SageMaker: Our biggest technical win was successfully configuring a SageMaker Docker file. Getting the environment to run correctly with our specific AI models was a complex puzzle, but seeing it finally deploy was a massive relief.

The "First Block" Moment: Seeing the very first AI-generated structure pop into the server—proving our pipeline actually worked—was an incredible feeling.

What we learned

Cloud Infrastructure: We went from knowing the basics to understanding how to create and manage on-demand instances on AWS.

Pipeline Orchestration: We learned how to chain distinct AI models (Titan and HunYuan) together into a single, functional product.

What's next for AI Minecraft Builder

The Spellbook: Adding a "Library" feature where users can save their generated models to spawn them again later without re-generating.

Scaling Spells: Implementing arguments to the command to allow users to adjust the size and scale of the model before it generates.

Material Matching: Improving the voxelizer to better match the colors of the AI model to specific Minecraft block textures (e.g., using "Oak Planks" for brown pixels).

Structure Production Speedup: Refining the voxelization algorithm and server-side placement logic to speed up the physical building of large structures.