Custodian

An autonomous data steward for DataHub. It investigates broken data, retires dead tables, and writes every finding back into the catalogue; so the next person inherits instead of repeating the work.

Comment

Track: Agents That Do Real Work

Inspiration

Every data team has the same two unglamorous jobs, and nobody wants either.

Something breaks, and a person spends an afternoon clicking through lineage graphs to work out what else broke and why. Then, separately, thousands of tables quietly rot because retiring one means proving nobody depends on it — and nobody has time to prove a negative.

Both jobs are reading a graph and reasoning about it. That's the thing an agent with a metadata catalogue can actually do well.

But the thing that bothered me most: when a human finally does that work, the answer usually lands in Slack and evaporates. Six weeks later someone investigates the same incident from scratch. The catalogue is right there — so Custodian writes every finding back into it. That's the whole idea. The output isn't a chat message; it's durable metadata the next person, or the next agent, will find.

What It Does

custodian investigate <table>

Give it one broken table. It walks lineage in both directions, reads what each asset says about itself, and produces a post-mortem: blast radius, root cause with quoted evidence, owners to notify, recommended actions. Then it saves that post-mortem as a document in DataHub and tags every impacted asset.

The evidence discipline matters. An earlier version inferred the cause from the shape of the lineage graph and confidently sent you to debug the wrong system. It now reads asset descriptions and must quote the line supporting its conclusion. On our demo it finds an engineer's own migration note two hops upstream — and correctly recommends fixing the extractor and backfilling, rather than debugging the dashboard where the symptom appeared.

custodian sweep

Finds tables nobody depends on. Scoring runs in plain Python across the whole estate (no downstream lineage $0.40$, no owner $0.25$, no description $0.15$, stale $0.10$, no queries $0.10$) with orphanhood as a hard veto. The model never does the arithmetic; it spot-checks the shortlist for intent, which is the part that needs judgment.

It knows when not to act. A retention or legal-hold property vetoes retirement outright. On the showcase estate, both genuine orphans score $0.80$ — dead by every measure — and both are spared, because nothing reads a regulatory archive; that's what one is for. They're reported but not dropped, so a human sees they were considered.

custodian approve

Custodian cannot deprecate anything. It stages candidates with reasoning. A human runs approve, and only then does a real deprecation land. The agent proposes; a person disposes.

custodian watch

The sweep on a loop, using DataHub itself as its dedup store, so it's stateless across restarts. It also retracts: an asset staged last week that has since gained a consumer gets unstaged, because a stale candidate is dangerous — someone might approve it.

Notifications

--notify renders digests to owners of affected assets. Rendering requires both an SMTP config and an explicit --send.

How We Built It

Python, the DataHub MCP server over stdio, and Gemini's free tier.

The agent loop is hand-rolled. The SDK's automatic function-calling path crashes on pickle (_asyncio.Task), and working around it hit a second failure from an MCP 2.0 field rename. Owning the loop fixed both and made everything after it possible: per-command tool allowlists, result caching, token pacing, and — most importantly — the ability to detect failed writes.

Deprecation is the one operation that bypasses the agent entirely, going through the CLI directly. That split is deliberate architecture, not convenience: the agent has no tool that can retire anything.

Four interchangeable providers (Gemini, Groq, Cerebras, OpenRouter) behind one protocol.

Challenges We Ran Into

The DataHub MCP server reports write failures as ordinary result text with isError unset. A model reads "Error calling tool…" and cheerfully reports success. This bit us four separate times — non-existent tag URNs, bare property names instead of full URNs, malformed property values besides. Custodian now inspects result text itself and prints a warning naming the failed calls. Every claim in this submission was verified by reading DataHub back, never by trusting the agent's narration.

Two silent-truncation bugs, both worse than crashes. sweep defaulted to scanning 25 assets of 68 — in arbitrary catalogue order — and then reported "no dead assets found." A confident all-clear from a partial scan is more dangerous than an error. And the retention veto initially checked the wrong field, so it protected nothing while appearing implemented; a table marked "retain indefinitely" was staged and retired. Both are fixed, and coverage is now stated explicitly.

Free-tier limits shaped the architecture, for the better. Hitting the 250K input-tokens-per-minute ceiling forced per-command tool allowlists ($19{,}115 \to 7{,}163$ tokens), result caps, and moving the mechanical census out of the model entirely. That last change cut a sweep from minutes to seconds.

Accomplishments We're Proud Of

  • The spare behavior. Building an agent that finds dead tables is straightforward; building one that scores them $0.80$ and then refuses to touch them because of a retention policy is the harder and more useful thing.
  • Every write is verified in DataHub, not asserted. examples/ contains real generated output, pulled verbatim out of the catalogue.
  • It runs on a free tier with no credit card, so a judge can clone it and run it for $\$0$.

What We Learned

That the interesting engineering question isn't "can the model do this" but "which half of this should the model do." Enumerating lineage across 68 datasets is deterministic work that a model does slowly, expensively, and unreliably. Judging whether an orphaned table is abandoned is the part that needs judgment. Every significant improvement came from moving work out of the model and leaving it the judgment.

Also: an agent that writes to a production system needs to distrust its own tools. Nearly every real bug was a write that silently didn't happen.

What's Next

  • custodian explain — generate descriptions for undocumented assets, attacking the cause of one of its own deadness signals
  • Learning from approve decisions: every yes/no is free ground truth on the scoring
  • More vetoes — "referenced in a dbt project," "consumed outside the catalogue"
  • Conversation compaction for long investigations

Built With

  • cerebras
  • datahub
  • google-gemini
  • groq
  • mcp
  • openrouter
  • python
  • rich
  • typer
Share this project:

Updates