DevGuard Enterprise

A closed-loop incident agent for DataHub: proves root cause from the lineage graph, refuses when the evidence is insufficient, and writes verified incident knowledge back as first-class metadata.

Comment

Inspiration

A column got renamed upstream. Three dbt models broke. A churn model kept training on a stale feature table and nobody noticed for a day.

The error message told us what failed. It could not tell us which change caused it, what else it touched, who owned the table, or whether the obvious fix was safe. Every one of those answers already existed — in the data catalog. Nothing was reading them.

That is the gap DevGuard was built for. Not "an AI that explains errors," but an agent that treats DataHub as the reasoning substrate: proves root cause and blast radius from the graph, routes approval to the asset's real registered owner, verifies recovery, and then writes what it learned back into the catalog as first-class metadata — so the next incident on a related asset starts from more knowledge than the last one.

The loop closing is the whole thesis. A catalog that only gets read grows stale. A catalog that gets written by every resolved incident compounds.

What it does

Nine agents, each with one responsibility and an explicit tool allowlist enforced before a request reaches the MCP pipe:

Agent Does Reads from DataHub
Watcher Observes the real failure — exit codes, build output —
Cartographer Resolves a string in a log (stg_users) to a real URN, pulls schema truth search, get_entities, list_schema_fields
Archivist Has this happened before? search_documents, grep_documents
Pathfinder Column-level blast radius, incl. the ML model at the end of it get_lineage, get_lineage_paths_between, get_dataset_queries
Diagnostician Root cause — and holds zero tools nothing, by design
Surgeon Proposes a minimal diff on a branch. Never applies —
Referee Validates in a throwaway schema, then verifies recovery Assertions (read-only)
Magistrate Routes approval to the owner resolved from the graph get_entities (owners)
Scribe The only agent that can write, and only after recovery is verified 5 mutation tools + GraphQL

Five artifacts land in DataHub, idempotently, only once recovery is verified:

  1. Incident raised, then resolved — raiseIncident → updateIncidentStatus
  2. Post-mortem runbook — save_document → Context Document
  3. Column-level tag + description on the schema field
  4. Structured incident properties
  5. Ownership signal

And the loop closes. On the second pass, the Archivist retrieves four runbooks the first pass wrote — straight out of the catalog. That is the moment worth watching.

A reviewer can see all of it with zero infrastructure: cd frontend && npm ci && npm run dev → /command. No DataHub, no database, no API key, no backend, no Python. The UI replays committed proof packs.

How we built it

DataHub is reached over the real MCP protocol. backend/v2/datahub_client.py spawns the official mcp-server-datahub@0.6.0 as a subprocess and speaks JSON-RPC 2.0 over stdio — not an HTTP envelope shaped like MCP. The server's own initialize / tools/list / tools/call handshake is what runs, and agents construct calls against the live inputSchema the server returns, which is why tool contracts are assertable in unit tests with no server running.

The substrate is real. PostgreSQL 16 → dbt Core 1.12 → a scikit-learn churn model, ingested into DataHub with real recipes (recipes/postgres.yml, recipes/dbt.yml, a business glossary, structured-property definitions). The incident is a real column rename that really breaks a real dbt build.

Three design rules everything else follows from:

1 · Refusal is structural, not prompt discipline. The Diagnostician has no tools. An evidence chain must carry at least one RUNTIME item and at least one DATAHUB_GRAPH item before a root cause is permitted — runtime alone is just an error message; graph alone is a theory about an incident that may not have happened. When the chain cannot form, the agent returns INSUFFICIENT_EVIDENCE and names the missing class. d5-refusal is that refusal, recorded, with its own proof pack.

2 · Every claim has an artifact behind it. Each run emits a proof pack: every tool call's exact request and response, every evidence item with its provenance, every handoff, every write-back payload and the catalog's reply. The UI reads its numbers out of those files, so what a judge sees on screen and what is on disk cannot diverge. Anything unmeasured renders N/A with the reason attached — never a plausible-looking zero.

3 · The docs are checked by tests. tests/test_judging_matrix.py fails the build if a cited artifact path stops resolving or a quoted figure is overstated. tests/test_upstream_claims.py fails if a "filed upstream" checkbox is ticked while the issue is unfiled. tests/test_least_privilege_claims.py fails if a security doc claims a privilege denial the evidence does not prove — added because it once had.

Observability is OpenTelemetry → SigNoz v0.135.0: one incident is one distributed trace, spans carry agent name, evidence IDs consumed, decision taken, and log↔trace correlation. scripts/verify_otel.py stands up an in-process OTLP receiver and asserts against decoded protobuf, so the telemetry pipeline is proven in CI without SigNoz running.

Challenges we ran into

Five live-server behaviours where the obvious implementation is wrong. Each cost real hours; all are documented as integration findings rather than hidden:

  1. Aspects split across sibling entities. DataHub models a warehouse table and its dbt node as siblings and merges them in the UI. GraphQL does not. Profiling lands on one URN; ownership and assertions on the other. Our capability prober reported "no owners" on a catalog the UI was plainly showing two owners for. It now follows siblings and records which URN answered.
  2. MLModelProperties.trainingData produces no traversable graph edge. Every blast radius stopped at the mart and silently never reached the churn model it would break — the single most important hop in the demo, missing without an error. Fixed by modelling the training run as a dataJob: dataset → dataJob → mlModel.
  3. A tag must exist before add_tags can apply it. Our first live write-back failed artifact 3 with Failed to validate label … Urn does not exist. We kept the behaviour rather than working around it — an agent that can invent vocabulary can invent meaning — and made tag creation an operator responsibility.
  4. A structured property whose string value parses as a URN breaks searchAcrossLineage, with nothing connecting the failure to the write that caused it. We store a bare id now.
  5. get_lineage defaults to max_results=30 and truncates silently. That capped every blast radius until commit 34639d4. Our bug, not DataHub's — but the failure mode is a plausible wrong answer in exactly the use case where you least want one.

The one that was a security lesson. Our least-privilege verifier reported 9/9 passing against a stock quickstart — before the policy was correctly scoped. A control that passes before you configure it is not passing. The quickstart ships METADATA_SERVICE_AUTH_ENABLED=false, under which nothing evaluates policy. And DENY cases are mutations: they were not refused, they executed. That run soft-deleted the dataset under test, put a cycle in its lineage, and created a policy granting the test account MANAGE_POLICIES — the suite escalated its own privileges as a side effect of testing that it couldn't. All repaired. The failing run is kept verbatim next to the passing one (ALLOW 5/5 · DENY 7/7 with auth on), because the failure is the finding, and the verifier now refuses to run against an unenforcing server.

The limitation we could not engineer around. The capture environment blocked egress to our inference endpoint. Every recorded run carries model=null, tokens=0, and root causes are derived deterministically from runtime evidence. The UI states this in full rather than rendering a zero. The evidence rule, the refusal path and chain validation are proven; the quality of model reasoning is not. It is the first thing in our Limitations section, not the last.

Accomplishments that we're proud of

  • DataHub v1.7.0 stood up and interrogated, not asserted. 27 capabilities probed against a running instance: 25 verified · 2 present-but-empty · 0 absent · 0 error, with every raw GraphQL response kept. The matrix has four statuses because collapsing "is the field in the schema" and "did this catalog return data" into one supported column is how a capability matrix starts lying. The two empties are freshness and usage stats, which need a connector that reads warehouse query history — DataHub's Postgres source doesn't, so filling those panels would mean inventing numbers.
  • 1,104 tests that run with no API key, no network and no catalog.
  • A published negative result. Our retrieval ablation made time-to-root-cause slower (5.14 s vs 4.87 s, N=5 per arm). Published anyway, because it was measured.
  • A withdrawn upstream proposal, kept on the record. We had drafted a request for a truncation marker on lineage responses. Reading a live response showed total already sits beside the results page. The information existed; we weren't reading it. A withdrawn proposal is a result.
  • 23 real screenshots of the running catalog, including the post-write-back state — the incident, the column annotation, the structured properties, the governance tab.

What we learned

A catalog is not a data store — it is the only place several answers exist. Grep cannot tell you which dashboard consumes a renamed column. Config cannot tell you who owns a production table today. The graph can, and that is what makes DataHub the right substrate for an agent that has to prove things rather than assert them.

Agent governance has to be structural. We started with prompt wording and ended with architecture: the agent an injection would most want to influence is the one we gave zero tools. Allowlists sit in front of the transport, so a violation is a Python stack trace, not a server-side rejection to interpret.

Honesty is a feature, and it is testable. Every limitation above is in the README, some of them load-bearing. We wrote tests that fail the build when the documentation flatters the code. The most useful artifact we produced all hackathon is a failed verification run we chose to keep.

What's next

  • File the two upstream findings. Both are verified against master @ f4fda77c with copy-pasteable reproductions, and finding 01 has a complete patch that applies cleanly. Unfiled for two stated reasons: the duplicate search, and the Java half was never compiled from our sandbox. Filing a confidently-wrong patch costs a maintainer more than filing nothing.
  • A model-backed run, to close the one limitation the architecture is already built for.
  • MCP parity for incidents and assertions. Today the write-back drops to raw GraphQL for one of five artifacts because incidents aren't exposed over MCP, which means the least-privilege boundary is implemented twice — and a governance boundary implemented twice is one with a hole in it.
  • A write path for assertion results. DataHub OSS has no reportAssertionResult, so DevGuard corroborates assertions without ever authoring one. For a team letting agents remediate, the verification outcome is the single most important thing to record.

Built With

Share this project:

Updates