WorldEval

Solo Multi-Action Construction — A verified participant turns, walks, gathers, carries, deposits, builds, and celebrates in one sealed run.
Mini RTS: Bridge Battle — Independent agent plans collide as both factions contest the central bridge.
Labyrinth Run: Verified Result — The final ranking exposes finish time, travelled cells, dead ends, and path efficiency.
Mini RTS: Verified Victory — Terra’s surviving militia hold the battlefield after destroying Luna’s stronghold.
Artifact Replay — Reconstruct every round alongside faction state, recorded events, and the objective timeline.
Evidence Podium
Labyrinth Run — Sol, Luna, and Terra navigate identical maze lanes using locally visible passage choices.
Solo Controller Lab concept — Configure a model, run a construction task, and inspect every intent, state change, and evidence.
Labyrinth Run in the Controller Lab — Three models race the same maze with private observations and no shared vision.
WorldEval — Evaluate what intelligent agents do, not only what they say.
Crossroads Conquest: Centre Conflict — Sol’s siege breaks through Terra’s defenses while Luna scouts for an opportunity.
GIF
Crossroads Conquest local demo — Three factions gather, fortify, and contest a shared strategic world.

Inspiration

WorldArena began unexpectedly.

At a Wednesday builders event in London, we met when one of us invited the other to join a hackathon that Saturday. It was one of our first hackathons. We ended up in a room with around five people, exchanging ideas and challenging each other’s assumptions.

One thought kept returning to us:

If something takes us a week to code today, there is a real chance it will be one prompt away in six months.

That made us ask a different question: what will not be solved by simply making language models better at generating text?

Modern LLMs are becoming extraordinarily capable, but they can still fail surprisingly simple problems that require grounding, situational awareness, and an understanding of consequences.

A striking example is the petrol-station question:

“My car needs fuel and the petrol station is five minutes away. Should I walk or drive?”

A chatbot might recommend walking because it is healthier and environmentally friendly—while completely missing that the purpose of the journey is to take the car to the petrol station.

The model can produce fluent advice while failing to connect the goal, the object, and the required physical action.

That gap becomes much more important when AI moves beyond chat. Future models may operate software, vehicles, robots, infrastructure, or other systems where actions change the world. They may also need to cooperate with unfamiliar agents, respond to deception, recover from mistakes, and behave safely when another agent—or a human—is acting unpredictably.

We therefore need to evaluate more than what a model can say. We need to evaluate what it does, what happens next, and whether its decisions actually work.

That idea became WorldArena.

What WorldArena does

WorldArena is a deterministic 3D research environment for evaluating LLM agents through observable behaviour.

Multiple independently configured agents receive partial information about a persistent simulated world. They must plan, gather resources, construct buildings, manage supply lines, negotiate, form agreements, adapt to opponents, and live with the consequences of their decisions.

The LLM acts as the strategist, but it does not directly control coordinates, physics, damage, resources, or scoring. Those remain under the authority of the deterministic Godot simulation.

This separation is important. A model cannot simply claim that it completed an objective. Its plan must produce valid actions, and those actions must create verifiable changes inside the world.

WorldArena records those changes as evidence: accepted actions, simulation events, state hashes, resource usage, outcomes, and replay artifacts. The goal is to evaluate behaviour without relying on another LLM to judge whether the first LLM performed well.

The larger vision is to create multiple games and environments that expose different behavioural qualities:

Spatial and situational reasoning
Long-horizon planning
Recovery after failure
Resource management
Cooperation and negotiation
Trust, deception, and betrayal
Delegation between agents
Tool use and protocol compliance
Reliability under limited information
The relationship between an agent’s stated plan and its actual results

These environments could eventually help us understand what information models need, which controller interfaces work, where models consistently fail, and what new training data could improve them.

How we built it

We approached the project as two connected systems: an LLM controller and an authoritative game environment.

The Python and FastAPI layer manages model calls, private observations, validation, budgets, typed actions, session isolation, and replay artifacts. Agents submit structured high-level plans rather than directly manipulating the game.

Godot owns the physical truth of the world. It resolves movement, gathering, construction, combat, territory, and victory through deterministic simulation rules. Rendering and animation make the outcome understandable, but they cannot change the result.

The architecture is roughly:

LLM agents
    ↓
FastAPI controller and action validation
    ↓
Godot controller and task verification
    ↓
Deterministic world simulation
    ↓
Replays, state hashes, events, and evaluation evidence

A React and TypeScript dashboard provides the controller interface, readiness checks, match configuration, and result views. The project also includes automated tests, headless simulation runners, replay verification, and a GitHub Pages project site.

Challenges

The biggest challenge was that we are not game designers.

We had to learn Godot, GDScript, navigation, game-state management, asset integration, textures, cameras, and deterministic simulation while also designing the evaluation system itself.

Creating fair game rules was harder than making the world look good. Early rules could produce boring strategies, impossible objectives, or behaviour that did not reveal anything meaningful about the model.

Another difficult decision was choosing the correct level of control for the LLM.

Should the model receive a broad objective such as:

“Gather wood from that tree.”

Should it issue a milestone such as:

“Go and break down the tree.”

Or should it control atomic actions such as:

Walk to the tree.
Equip the axe.
Strike the tree.
Collect the wood.
Return the resources.

Too much abstraction hides important failures. Too little abstraction turns the benchmark into a test of command formatting rather than planning. Finding a useful boundary between model intent and deterministic execution required repeated redesign.

We also learned that asking an AI system to “decide everything” does not automatically produce good product or benchmark design. The model could generate large amounts of code, but it could not replace careful decisions about authority, rules, incentives, action granularity, failure states, or what should count as evidence.

We had to work closely with the model: questioning its decisions, testing the results, changing the interfaces, and repeatedly simplifying parts that had become unnecessarily complicated.

What we learned

The most important lesson was that benchmark design is as much about interfaces and incentives as it is about models.

An agent’s performance can change dramatically depending on what it observes, how actions are represented, how much control it receives, and whether failures produce useful feedback.

We also learned that deterministic execution is essential. If the environment, evaluator, and model all have overlapping authority, it becomes difficult to know what actually caused an outcome. Keeping the LLM responsible for strategy and the simulator responsible for physical truth creates a much clearer evaluation boundary.

Finally, we learned that building with AI still requires active human judgment. The model made it possible to explore a much larger project during one of our first hackathons, but the strongest results came when we treated it as a collaborator rather than an automatic decision-maker.

What is next

WorldArena is currently a working research prototype, but the idea is designed to expand.

The next steps are to:

Add more games and evaluation environments
Test different observation and controller formats
Compare models across deterministic seeds and starting positions
Expand adversarial and social scenarios
Improve behavioural metrics and evidence visualisation
Generate replay-backed datasets from successful and failed strategies
Run larger benchmark seasons
Host the controller and interactive demos publicly

The long-term goal is not only to identify which model wins a game. It is to understand why an agent succeeded, why it failed, and what its behaviour tells us about the next generation of AI systems.

WorldArena evaluates what an LLM does—not only what it says.