-
-
A live run on the covered profile. Eight consumers found across three platforms, four repairable here, executed before anything is written.
-
The synthetic estate builds from scratch: six raw tables, ten dbt models, and a pandas transform no SQL parser can read.
Inspiration
Ask DataHub what's downstream of a column and get back an empty list. That means one of two things: nothing depends on it, or nothing that depends on it is visible. There's no field in the response that tells you which.
A data engineer absorbs this without noticing — they see an empty result and think "right, but we never connected Looker." That knowledge lives in their head and nowhere in the response. An agent has only the response, and the default reading is that empty means safe.
For a read-only agent that's survivable. A wrong SQL answer costs thirty seconds. But an agent writing code into a repository will confidently fix the four consumers it can see while three others break silently after merge, and that code runs nightly until someone notices.
We wanted to know whether an agent could be built that knows the difference.
What it does
Warden reviews a proposed change to a data pipeline and:
- Resolves what the change refers to in the graph, flagging ambiguity rather than guessing
- Computes a coverage score — deterministic arithmetic over reachable entities, platforms without lineage connectors, and parsed-versus-inferred edges
- Classifies breakage across downstream consumers
- Generates dbt fixes, runs them with
dbt build, and opens a PR - Or declines to generate anything, names the missing metadata, and writes the blocked decision into DataHub as a queryable Decision document
The same change against a well-covered graph produces a PR with four verified file edits. Against a graph missing its BI connector, it produces a refusal naming the dark platform. Same code, different graph.
How we built it
Six agents, decomposed by failure mode rather than by task step. Each exists because something goes wrong there that nothing else catches — retrieving irrelevant context, reading an empty result as evidence of absence, opening a PR containing code that never compiled.
Every read and write goes through mcp-server-datahub. No direct GraphQL, no SDK calls from any agent. Where the MCP surface couldn't express something, we reported it as a limitation rather than routing around it.
The coverage computation deliberately contains no LLM. That's what lets verify.py prove the central claim — this graph produces a refusal — in CI with no model, no network, and no running catalog.
DataHub has no native way to record what it isn't connected to. Structured properties aren't supported on dataPlatform entities, so we materialise a registry dataset per platform carrying connector status as custom properties. The auditor reads that back over MCP and uses it as the coverage denominator — an agent measuring "how much of what I retrieved did I understand" always reports complete coverage.
Stack: Python 3.11, Pydantic, mcp-server-datahub, DataHub OSS 1.7.0, dbt-core on DuckDB, pytest.
Challenges we ran into
The gate was wrong for two batches. The original condition blocked only when coverage was thin and nothing had been found — so a rename against a dark graph sailed through, because it had found real breakage. Every unit test passed. It surfaced only when we ran the same change against both graph profiles side by side.
The reasoning that fixes it: a confident impact list from an incomplete graph is still partial. Finding something isn't finding everything, and only the latter licenses writing code.
Verification also took a second pass. Testing generated fixes alone compiles them against a source that still has the old shape, so the compiler reports the fix as broken when the change it anticipates simply hasn't happened yet. What has to be verified is the change and its fixes together, because that's what a merge produces.
Environment: DataHub's quickstart hardcodes metadata service auth off, and re-downloads the compose file on every run, so an env var won't override it and editing it in place doesn't survive.
What we learned
A gate that refuses everything teaches people to ignore it. Widening a type can't break a reader whether or not that reader is visible — its safety doesn't rest on having looked. Blocking it would be refusing work for no gain. There's a real distinction between safety by absence of evidence and safety by construction, and conflating them makes the gate either too permissive or too noisy.
Not every gap blocks. A platform with no connector that holds only raw source tables is a real gap, but it can't conceal anything downstream of a change.
Coverage gates safety claims but not completeness claims. Warden can prove a consumer exists without proving it found them all, so impact lists say "at least N" and name the dark platform.
The generated documents are the product. A reviewer spends most of their time inside the PR body or the refusal message. Those have their own tests, because prose regressions are otherwise invisible.
What's next
Calibrating the coverage threshold against measured outcomes rather than judgment. Weighting dark platforms by consequence rather than entity count — losing four dashboards costs 4/27 of the count and 100% of consumer-facing visibility. Watching the metadata change stream so held decisions resume automatically. Multi-repository remediation, since downstream fixes often live in repos the change author doesn't own.
Log in or sign up for Devpost to join the conversation.