GenCraft

Inspiration

We were inspired by the rapid growth of generative AI and how it can unlock new creative tools for people. Minecraft felt like the perfect environment to explore this idea because it’s fundamentally about building and creativity.

We asked ourselves: what if you could describe something in words and have it instantly appear in Minecraft?

We were also inspired by other creators experimenting with AI-generated tools and worlds, especially work from Falcraft. Seeing how AI could interact with Minecraft environments motivated us to try building something ambitious ourselves — a system that could turn a simple text prompt into a fully built structure in-game.

What it does

GenCraft is a Minecraft Forge mod that generates 3D structures directly in the game from a text prompt using AI.

Players can type a command like: /generate generate 32 castle and the mod will automatically generate a castle and build it in front of the player.

The system converts a natural language prompt into a 3D model, voxelizes it, and maps the colors to Minecraft blocks to recreate the structure inside the world.

This allows players to instantly generate buildings, statues, objects, or creative structures without manually placing thousands of blocks.

How we built it

We built GenCraft using Java and the Minecraft Forge modding framework.

The generation pipeline works in several stages:

1. Text → 3D Model

We use the Tripo AI API to convert a text prompt directly into a GLB 3D model.

2. Model Parsing

A custom GlbParser extracts the geometry and texture data from the GLB file.

3. Voxelization

The mesh is converted into a 3D voxel grid, representing the model as small cubes.

4. Color Mapping

Each voxel’s color is mapped to the closest Minecraft block using CIE-LAB color space, which helps match colors more accurately with the available Minecraft blocks.

5. World Generation

Finally, the blocks are placed into the Minecraft world in front of the player.

We also implemented a local caching system so that previously generated prompts reuse stored models instead of calling the AI API again. This speeds up generation and reduces API usage.

The mod uses a provider-based architecture, allowing us to easily swap between different AI providers such as Tripo, HuggingFace, Replicate, and Fal.

The core of this pipeline was inspired to us by Falcraft.

Challenges we ran into

One of the biggest challenges was converting AI-generated 3D models into Minecraft blocks.

The models produced by AI are smooth meshes, while Minecraft uses discrete cubes. Translating between these two representations required building a voxelization system that could approximate shapes while keeping structures recognizable.

Another challenge was color matching. Minecraft has a limited set of blocks, so we implemented a color-matching algorithm using CIE-LAB color space to find the closest block for each voxel.

We also ran into performance challenges when generating large builds. Some structures can contain tens of thousands of blocks, so we had to balance detail, speed, and stability.

Finally, integrating multiple APIs and handling generation failures required designing a flexible pipeline with fallbacks.

One of the problems we are still facing is being able to model structures from inside as well and not only just the outside.

Accomplishments that we're proud of

We're proud that we were able to complete such an ambitious idea.

Building a system that connects AI generation, 3D model processing, voxelization, and Minecraft modding was technically challenging, but we successfully created a working end-to-end pipeline.

Typing a prompt and watching a structure appear in the game is incredibly satisfying, and it shows how generative AI can enhance creativity inside games.

We're also proud that the system is modular and extensible, making it possible to plug in different AI providers in the future.

What we learned

Through this project we learned a lot about:

Minecraft Forge modding
Working with 3D model formats like GLB
Mesh processing and voxelization
Color matching using CIE-LAB color space
Designing modular software architectures
Integrating external AI APIs into real-time applications

We also learned how difficult it can be to translate AI outputs into structured environments like Minecraft.

What's next for GenCraft

In the future, we would like to expand GenCraft with several new features:

We want to train our own model to be able to generate and understand minecraft specifically
Be able to build structure from the inside
Better structure accuracy and improved voxelization
Support for larger and more complex builds
More intelligent block selection (using structural blocks instead of only color matching)
Real-time preview of generated models
Multiplayer support so players can generate structures together
A graphical interface instead of only commands

Our goal is to turn GenCraft into a powerful creative tool where players can bring their ideas into Minecraft instantly using AI.