MineGPT — Skip the Blank World. Finish It Together.

Inspiration

Creative sandbox games are at their best when friends turn a shared idea into something real. However, two kinds of friction often get in the way: setting everything up before anyone can play, and deciding what to create after everyone enters an empty world.

We wanted to explore a simple question:

Could AI help a group move from “What should we build?” to a shared plan without taking away the experience of building it?

We did not want MineGPT to be a one-click generator that creates a finished castle while the players watch. We wanted AI to behave more like an architect’s sketchbook: interpret an idea, propose a bounded plan, and make the next steps understandable.

The players should still decide whether to use the proposal, place every block, adapt the design, and experience the satisfaction of finishing it together.

That became MineGPT’s central design principle:

$$ \text{GPT-5.6 proposes} \rightarrow \text{deterministic code validates} \rightarrow \text{the creator approves} \rightarrow \text{friends build} $$

MineGPT is our project, created specifically for the OpenAI Hackathon. It is an independent project and does not claim endorsement by OpenAI.

What MineGPT Does

MineGPT is a browser-native multiplayer voxel world designed for small groups of friends. Players can create an account and mine or place blocks together in real time.

Its defining feature is Worldsmith.

A player travels beyond the protected spawn town, describes a structure in one sentence, chooses a compact or shared size, and selects between two and eight available materials. MineGPT sends this bounded build context to the OpenAI Responses API, where GPT-5.6 produces plain-language construction stages and a structured spatial proposal.

The model’s response is never treated as executable authority. A deterministic compiler expands and validates the proposal against:

The selected material palette
The permitted local build envelope
Protected areas and existing room edits
World and placement limits
Terrain support and foundation coverage
Structural connectivity
Five permitted spatial-operation types
A maximum compiled plan size of 256 placements

The available operations are deliberately constrained to filled boxes, hollow boxes, lines, columns, and gable roofs. This gives GPT-5.6 enough flexibility to design meaningful structures without granting it unrestricted control over the world.

If the proposal passes validation, its creator receives a translucent ghost blueprint, a required-material list, and a staged building guide. The result begins as a private draft. Publishing requires a separate authenticated action by the creator, and publication still does not place any blocks.

Friends construct the blueprint manually. Matching accepted edits update shared progress for everyone in the room, while the active blueprint and construction state persist across reconnects.

Beyond Worldsmith, MineGPT includes deterministic terrain, 64 playable materials, eight climate-driven biomes, ten code-native voxel creatures, mining, crafting, trading, brewing, enchanting, smelting, weather, structures, and signal circuits. We distinguish these broader gameplay systems from the room-authoritative state used for multiplayer building.

How We Built It

The MineGPT client is built with React, TypeScript, Vinext, Three.js, WebGL, and a custom voxel runtime.

We implemented the core game systems directly in the browser, including:

Deterministic terrain generation
Chunk generation and meshing
Grounded player movement and collision
Block targeting, mining, and placement
Lighting and weather
Voxel creatures
Inventory and progression systems
Worldsmith ghost-blueprint rendering

The multiplayer infrastructure runs on Cloudflare. A public gateway Worker provides a stable REST and WebSocket origin and connects through a service binding to the stateful backend Worker.

The backend uses three SQLite-backed Durable Object roles:

PlayerAccount manages credentials, expiring sessions, generation allowances, and single-use play tickets.
RoomDirectory assigns rooms to their corresponding shards.
GameRoom owns connected players, accepted block edits, reconnect state, Worldsmith drafts, the published blueprint, and shared construction progress.

Worldsmith uses the OpenAI Responses API with GPT-5.6, strict structured output, store: false, server-side credentials, and a hashed safety identifier. The model receives only the constrained information necessary to create the proposal, such as build intent, size, direction, selected palette, and local coordinate boundaries. It does not receive account credentials or unrestricted world data.

A completed response that fails schema or compiler validation may enter a bounded repair path. Transport failures, refusals, incomplete output, and model mismatches fail safely instead of being misrepresented as ordinary validation errors. Time limits, rate limits, and daily allowances protect the hosted service, while immutable request snapshots allow safe replay without automatically making another model request.

We also used Codex with GPT-5.6 Sol as an engineering collaborator. It accelerated implementation and troubleshooting across the portal, Three.js runtime, network client, OpenAI adapter, Worker routes, Durable Object persistence, validation, automated testing, and guarded release workflow. Product direction and final acceptance remained under human control.

Challenges We Faced

The hardest challenge was transforming probabilistic model output into safe, useful geometry.

A response can be valid JSON while still describing unsupported materials, disconnected shapes, conflicting placements, insufficient foundations, or structures outside the permitted space. Solving this required treating every model response as an untrusted proposal and building a deterministic compiler around it.

We also encountered a real strict-structured-output schema incompatibility. Correcting the schema was only the beginning. We had to trace the complete workflow from request construction through model output, validation, bounded repair, draft storage, ghost preview, creator publication, peer delivery, progress tracking, and reconnect restoration.

Latency created another challenge. A design request may require model generation, schema validation, geometric compilation, and—in specific completed-response failure cases—a bounded repair attempt. We therefore implemented explicit call and workflow budgets rather than allowing uncontrolled retries.

Realtime authority was equally demanding. MineGPT needed to distinguish local client state from trusted room state, reject forged reconnect attempts, make play tickets single-use, restrict publication to the draft creator, preserve accepted edits, and prevent clients from directly mutating authoritative progress.

Release verification also mattered because the public application spans a frontend, gateway, backend Worker, and persistent Durable Objects. An HTTP success response alone cannot prove that the intended source is running. We added automated checks for source and build identity, production artifacts, Worker packaging, startup behavior, and the authenticated two-client Worldsmith protocol.

Accomplishments We Are Proud Of

We are most proud that MineGPT is neither a chat box attached to a game nor an automatic world generator.

GPT-5.6 contributes flexible language understanding and design proposals, while deterministic software and explicit human decisions retain authority. The model cannot directly mutate the world, publish its own draft, introduce arbitrary materials, or bypass the compiler’s spatial and safety rules.

In our recorded production evaluation, Worldsmith achieved 12 out of 12 mechanical validation passes across four prompts repeated three times. All 12 accepted plans produced unique compiled-plan hashes. Four required bounded repair, demonstrating that the repair path could recover from deterministic rejection without creating an uncontrolled retry loop.

An authenticated two-client production smoke also exercised proposal generation, deterministic rejection and repair, creator-only publication, delivery to a second player, matching-block progress, replay protection, reconnect restoration, logout, and session revocation.

These results establish important implementation properties. We do not present them as proof of visual quality, user preference, accessibility, product demand, or large-scale multiplayer capacity. Those questions require evidence from real participants and dedicated load testing.

What We Learned

The most important part of a generative product is often not the prompt. It is the authority model surrounding the response.

Strict schemas are useful, but deterministic validation, visible previews, explicit approval, bounded repairs, finite failure states, and carefully scoped persistence are what make probabilistic output usable in a shared environment.

We also learned to keep different forms of evidence separate:

Passing tests demonstrate behavior under tested conditions.
Matching release identities demonstrate deployment provenance.
A hosted two-client smoke demonstrates that the protocol is connected.
Mechanical evaluations demonstrate compiler and validation behavior.
Only real participant testing can demonstrate comprehension, enjoyment, and sustained value.

Most importantly, we learned that AI can support creativity without replacing human agency. In MineGPT, the model helps a group find a direction, but the players still create the world.

What’s Next

Our next step is paired usability testing with first-time participants. We plan to measure:

Time to the first valid Worldsmith plan
Understanding of the ghost blueprint
Time to the first matching block
Understanding of shared progress
Blueprint completion rate
Return intent after the first session

We also plan to create an updated browser demonstration of the complete Worldsmith journey, optimize the larger frontend bundles, expand touch-device validation, add account-recovery and moderation tools, test multiplayer capacity rather than inferring it from configuration, and carefully explore cross-shard coordination.

MineGPT’s principle will remain unchanged: