• 

• 

• 

• 

• 

Inspiration

Every game interface does two jobs: it finds the relevant numbers among everything the engine tracks, and it explains what they mean. That is exactly what a metadata catalog does for a data warehouse.

So we removed the interface. The city runs, it writes hundreds of thousands of rows to Postgres, and the player sees only the city and eight levers. No charts, no counters, no warnings. To find out what is happening, you query the warehouse. And if a catalog really does what an interface does, an agent with a good one should play better than an agent without.

That is a testable claim, which made it a benchmark rather than a game.

What it does

City Sim Agent Benchmark measures whether DataHub makes an AI agent better at its job.

A simulation puts a city through five years of deferred maintenance plus a fiscal shock. An agent gets twelve quarterly decisions and eight policy levers to bring a composite health index (solvency, satisfaction, service, population retention) back into the green.

The agent cannot see the city. It queries the warehouse. What varies between modes is only how much DataHub tells it about that warehouse.

Mode	Who decides	What they can see
human	You	The city render, the levers, and the Analytics Agent to ask questions
agent_raw	Our agent	SQL against the warehouse. No catalog
agent_datahub	Our agent	The same, plus DataHub descriptions, glossary and lineage
agent_datahub_live	Our agent	The same, plus assertions evaluated against the city each turn
agent_analytics	DataHub's Analytics Agent	It answers the questions instead of our agent

Two scripted policies bracket the range as fixed reference points, and they need no API key, so anyone can reproduce the endpoints of this benchmark for free in about a minute:

uv run blindcity run --mode good_policy --out results/good.json
uv run blindcity run --mode bad_policy  --out results/bad.json
uv run blindcity compare "results/*.json"

What we learned

The modes rank cleanly, and in the direction the whole premise predicted:

agent_raw < agent_datahub < agent_datahub_live < agent_analytics

More DataHub is better, every step of the way. But the sizes of those steps are the interesting part, because they are nothing like equal.

Descriptive metadata alone barely moved the needle. Adding descriptions, a glossary and lineage beat the uncatalogued control, but by little enough that we will not claim a size for it. Our table names are already human readable, so a description reading income_tax_revenue: Income tax collected tells an agent nothing the column name did not. If your naming is decent, documenting your columns is not what makes your warehouse agent ready.

Prescriptive metadata transformed behaviour. Assertions that document the operating range a healthy city holds to, and just as importantly which levers are not worth tuning, moved the agent from muddling through to something that looks like expert play. And it did so on less of everything: far fewer queries and fewer model calls. A mode that scored higher by querying more would just be a mode given more compute. This one stopped searching because it was told where to look. Describing your data says what a column is. Assertions say what good looks like, and that is the thing an agent cannot derive from the rows.

Handing the question to a purpose-built data agent beat handing our agent better metadata. DataHub's own Analytics Agent came out ahead of every mode we built, including the one with assertions. The lesson we take from that is uncomfortable and probably correct: the ceiling here is not set by how much metadata you can stuff into a prompt. It is set by how well the thing consuming it was built to consume it. A general agent with an excellent catalog is still a general agent. If you have a governed semantic layer, the agent that natively speaks it will beat the one you assemble yourself, and the effort is better spent governing the layer than tuning your own prompt.

Lineage matters most where the data cannot speak. Every lever is constant across the entire recorded history, so no amount of querying reveals that income_tax_rate drives income_tax_revenue. DataHub is the only place that relationship exists. Any correlation an agent could have found in the rows, it would have found without us. The catalog earns its keep on exactly the relationships the data cannot show.

There is a sharper version of the whole finding. Before assertions, in every run of every mode we built, no agent ever changed the income tax rate, while all of them fixed the roads. Roads announce themselves (wear = 0.97). A starved tax rate looks like a deliberate policy choice. Nothing in the data distinguishes "broken" from "intended", and that distinction is the catalog's job.

How we built it

• Simulation in pure Python, seeded and deterministic, roughly 820k rows for a 20-year run.
• Warehouse on PostgreSQL 16. Each run gets a private schema of views filtered to its own run_id, with search_path pointed at it, so run isolation is structural rather than remembered.
• DataHub Core via the OSS quickstart: schemas, descriptions, a 20-term business glossary, assertions, and 29 column-level lineage edges.
• Lineage is generated from the simulation's own causal graph, and every edge is validated against the running model. blindcity.sim.causal_check perturbs each source, runs the code that computes the target, and requires the target to move. "Derived from" states a demonstrated dependency, not a claim.
• DataHub holds the authoritative copy of the operating guidance. It is not baked into the agent's prompt. It is published to DataHub and read back at run time, so widening a band in the DataHub UI and re-running is a supported experiment. Publishing looks before it writes, and every run records which copy it acted on, plus a fingerprint, in catalog_source.

The comparison has to be honest to mean anything

That is most of the engineering.

• The three agent modes are one implementation instantiated three times. There is no if mode == anywhere in the loop, and the build fails if the modes' prompts, tools or turn messages differ by anything other than the catalog block. That is the only way "they differ in exactly one thing" is a property of the code rather than a promise in a document.
• No mode is told the scoring function. Not the index, not its components, not the threshold. An agent that knew the weights would optimise the metric instead of fixing the city.
• No city-state numbers appear in any prompt. Population, treasury and satisfaction are discoverable only through SQL.
• The viewer shows no numbers either. Worn roads look worn, unpowered buildings go dark. That is information parity with the agent modes, not a style choice.
• Every run starts from an empty warehouse, so no run inherits another's rows or planner statistics.
• Assertions are derived from the simulation's own mechanics and re-derived by a test that fails if a documented band stops matching what the code does. They state operating ranges and response times. They never name a lever to pull.

Challenges

Four bugs silently corrupted runs while every test passed. Each produced plausible numbers, which is what made them dangerous. The warehouse writer batched at 5,000 rows, so ticks (one row a month) stayed frozen while citizen_monthly trickled through, meaning the agent could not see the consequences of its own decisions. The glossary was emitted to DataHub but never linked to a dataset, so the catalog promised definitions and delivered none. Each turn opened on a blank conversation, so the agent re-ran information_schema 29 times and reissued the same fatal query on four separate turns. And a run that lost four queries to timeouts printed 0 errors.

The fix that mattered was not any single patch. It was recording the full transcript of every exchange: the system prompt, the conversation as the model received it, and every reply. Every one of these was invisible in the scores and obvious in the transcript.

A second runner drifted from the real one three times. The batch evaluator re-implemented the run loop, so it wrote no transcripts, could not clear a stale warehouse, and never printed degradation warnings. A batch could report a clean table while its runs were losing queries to timeouts. We deleted it. One runner, and a bash loop.

Our own assertions were wrong twice, and validation caught both. A satisfaction trend detector was exactly backwards, because in a collapsing city satisfaction rises: neglect drives a third of the population out and the survivors get shorter commutes.

For the judges

Branch main has not recieved an update since the competition close on August 10 at 5:00PM EDT. I have created a new branch called refinements. For full disclosure I finished this branch on August 11th after the competition deadline. I am mentioning it here since it refines some of the rough edges around the repo and has the most accurate results for all the agent benchmarks. It is considered the latest and greatest for this project. The README.md is clearer, the human mode has been tested and refined, and bugs were fixed in the agent benchmarks making the results of the benchmark more accurate. If you are interested, please view it on GitHub here.

I won't merge the PR to main, until after the competition has concluded.

Built With

• datahub
• docker
• fastapi
• openai
• postgresql
• psycopg
• pytest
• python
• sql
• uv