Gates of Galloo

Inspiration

In the HBO show Silicon Valley, Dinesh turns on a microphone inside a game and mentions pizza. The game immediately fills itself with Domino's advertising. It is funny because the world appears to be listening and rewriting itself around the conversation.

Gates of Galloo asks a more useful version of that question: what if speaking to a game did not merely open a chatbot or trigger an advertisement? What if the AI could understand the actual world, change it with you, and let you interrupt whenever its taste became questionable?

What it does

Gates of Galloo is a voice-directed voxel world shared by a person and an AI character named Nika.

The player can walk around the island and speak at the level of intention:

"Build a bridge in front of me."
"Create a domino path from the chapel to the harbor, but leave it standing."
"Extend the island with a mix of grass, stone, and water."
"I am hungry."
"That is too much. Undo it."

Nika reads the current state of the island and turns the request into visible actions. Terrain grows, objects arrive, characters move, weather changes, dominoes fall, and contextual storefronts can become part of the scene. These changes happen over time, so the player can watch, interrupt, redirect, or reverse the result.

The commerce interactions are no-charge previews. They demonstrate how a product, delivery, or storefront could enter a game as a playable event rather than as a detached banner.

Why WebMCP matters

The useful context is already inside the page: terrain, buildings, player position and facing, selected regions, actors, shops, domino runs, camera state, and recent changes.

Ordinary computer control would force an agent to infer all of that from pixels and simulate mouse clicks. WebMCP lets the website expose the world directly through meaningful actions.

Gates of Galloo registers six page tools:

inspect_world
list_assets
place_asset
erase_at
paint_path
undo_last_action

The agent works with island objects and validated grid coordinates instead of screen positions. The available vocabulary is intentionally defined by the game, while the model remains free to decide what to create with it.

This makes WebMCP part of the creative medium. The person and agent can see and change the same evolving artifact without repeatedly describing what is already on screen.

How it creates a better experience

The conversation stays inside the activity. The player does not have to export a screenshot, explain the map to a separate assistant, wait for generated code, and reload the result.

Instead, an idea becomes visible inside the current world. If the interpretation is wrong, the player can correct it while it is happening. References such as "here," "in front of me," or "beside the bridge" work because the agent receives the live player and scene state.

The result feels closer to improvising with another participant than issuing commands to a detached chatbot.

How we built it

The project began with the open-source Mykonos Island Voxels game by Ibrahim Boona. During the submission period, we transformed it into Gates of Galloo.

The browser adapts the live game controller into WebMCP tool definitions and registers them through document.modelContext.registerTool().

For voice interaction, OpenAI Realtime handles the live conversation and transcription. It has one high-level function called direct_world. That function sends the player's original request, together with a semantic snapshot of the island, to a server-side director built with the OpenAI Responses API.

The director produces a structured sequence of terrain, object, actor, camera, commerce, and domino operations. The browser validates that plan and performs it incrementally. The production interaction is model-directed rather than selected through keyword rules or canned responses.

We also added an embodied isometric player, pathfinding, bridge traversal, camera follow, open-ended terrain growth, mixed-terrain paths, live transcripts, interruption, conflict-aware undo, actors, weather, chain reactions, and contextual storefront scenes.

Challenges and lessons

The hardest problem was giving the model enough live context to make those actions feel intentional.

We had to solve placement validation, connected terrain painting, object footprints, isometric occlusion, bridge traversal, camera framing, world expansion, interruption, and undo without allowing one half-finished plan to corrupt the next.

We also learned that the design of the tool vocabulary changes the personality of the agent. A model with raw DOM access behaves like an operator. A model given verbs such as inspect, place, paint, stage, and undo can behave more like a collaborator.

Gates of Galloo is one playful example, but the same pattern can extend to design software, educational simulations, music tools, filmmaking environments, and any website where the human and agent should remain inside the work together.