Downstream Impact Guardian

When another team redefines a field, nothing errors — your models just quietly start returning wrong numbers. This agent reads DataHub on every PR and names whose assumptions break before merge.

Comment

Inspiration

I work on a data team, and we live in the middle of the blast zone. Upstream, the ingestion team ships a change: if it's a sudden schema change, our transformations fail loudly — annoying, but at least it pages someone. The worst case is quieter: a minor logical change to a field slips through, every query still runs, and the damage surfaces as incorrect measures in the downstream data layer. The assumptions broke; nobody had a data contract saying they couldn't. Weeks later an analyst or a business user notices a number that feels wrong, and someone gets to spend days tracing it back — dashboard, to mart, to staging, to ingestion.

And my own layer isn't innocent either: our changes break legacy dashboards downstream whenever someone forgets to tell the analytics team. Same failure, one seat over.

The repo told the truth the whole time: nothing in this repo broke. The repo just couldn't see the finance team's scheduled query, the ML feature pipeline, or the exec dashboard reading from that table. DataHub is the source of truth for that cross-system picture — what I wanted was an agent that cross-references it on every PR: not just schema, but columnar logic and semantic definition changes too.

That blind spot is structural. No amount of in-repo testing fixes it, because the blast radius of a schema or logic change lives across systems, and the only place that cross-system picture exists is the metadata graph. So I built the agent that stands in the gap: it reads the PR for what's proposed, reads DataHub for what's real, and refuses to let the two drift apart silently.

The insight worth sitting with: detecting a schema change is the easy half — a diff finds it, and in-repo CI can find it. Knowing who downstream depends on it is the hard half, and that knowledge lives in the catalog, not the code. Most data teams own a dbt repo, not the whole pipeline, so the cross-system consumer picture is exactly what they can't see from where they sit. This project is the shift-left half: catalog context (DataHub) substitutes for pipeline ownership, so any dbt team gets consumer-aware impact analysis, semantic-drift detection, and contract proposals — at PR time, before anything ships.

What it does

Downstream Impact Guardian is a reusable GitHub Action that catches breaking dbt changes that in-repo CI cannot see — because the evidence lives in DataHub's cross-system lineage, not in the repo. Its demo PR is deliberately green in normal CI; only the metadata graph reveals the damage.

On every pull request it:

  1. Detects three kinds of drift from three sources — schema changes (committed prod manifest vs PR manifest), business-logic changes (normalized-SQL diff of the same), and semantic drift (the PR's glossary YAML vs the live DataHub glossary).
  2. Measures real blast radius by cross-referencing DataHub lineage (searchAcrossLineage) with observed queries — cross-system, cross-team usage no single repo can see — and scores severity LOW/MEDIUM/HIGH/CRITICAL with a deterministic, inspectable rubric.
  3. Writes back to DataHub: a Data Contract for each impacted model, upserted with PROPOSED status. The catalog now durably records "this shape was promised" — approval happens by merging the PR, a human gate.
  4. Writes back to the repo: one idempotent PR comment with an ADK-narrated impact story (a real LLM call — Gemini or any configured provider; the demo uses openai/qwen3.6-flash) plus generated, mergeable compatibility code — *_compat / *_legacy dbt views and schema.yml tests that keep old consumers alive through the change.

Demo scale, honestly. The demo world is deliberately small: a four-model dbt project, and the downstream dashboard/query lineage in offline mode comes from mocked fixtures (clearly labeled in the report). Don't let the size hide the shape of the real problem. In production this is multiple repos, thousands of models, and Looker dashboards everywhere — and each team has access to only a few of those repos, so no one can see the whole graph from where they sit. That's precisely why the agent's judgment must come from the catalog and not the codebase: DataHub is the catalog center, the source of truth for what's live, and the historical snapshot reference for what changed. The agent's architecture is scale-independent — it reads whatever lineage the catalog holds, whether that's four models or four thousand.

The impact design, in four moves (the part reviewers should slow down on):

  1. Two orthogonal ratings. Severity is PR-level (LOW→CRITICAL, deterministic score — drives advisory vs strict). Impact level is per-consumer: 🔴 BROKEN / 🟠 DISTORTED / 🟡 ADVISORY, honestly labeled as the worst-case upper bound from the upstream change kind.
  2. Stakeholders come from the catalog. Each impacted consumer's DataHub owners appear right in the blast-radius table — and an unowned consumer is surfaced as a governance finding, never hidden. The informing protocol: comment always; Slack webhook on HIGH/CRITICAL; strict mode blocks CRITICAL; the Data Contract stays the durable record.
  3. Column-level evidence today. sqlglot expression diffing attributes logic changes to specific fields; observed queries are matched per column; a Column-level effects table shows facts, not inference.
  4. A precision ladder for consumer impact — declared > derived > worst-case. Declared is SHIPPED: consumers state what they read via depends_on_columns in their own dbt meta (ingested as custom properties); the guardian intersects with changed columns — match = BROKEN as fact, no match = 🟢 SAFE, the one verdict worst-case can never give. Derived (column-level lineage from the changed column's schemaField urn) is the roadmap rung; worst-case is always available. Each rung degrades honestly to the one below, and the agent audits declarations against observed queries — manufacturing the governance it consumes.

Two invariants make it trustworthy rather than magical:

  • DataHub only ever reflects reality. The PR is read locally as a hypothesis; nothing hypothetical is ever ingested into the graph.
  • The LLM narrates; it never scores and never authors merged code. Severity and codegen are deterministic and unit-tested. The agent's writes are proposals gated on human approval — 0 unattended production changes, by construction.

How we built it

  • Pipeline: Python 3.11, ~1,300 lines of core agent code, run entirely inside the consumer's GitHub Actions runner by a composite action (action.yml) — no hosted service anywhere. Any dbt repo adopts it with one uses: block; this repo dogfoods its own action.
  • DataHub integration, four distinct paths:
    • Agent Context Kit (datahub-agent-context[google-adk], build_google_adk_tools, read-only) gives the ADK narrative agent live lineage/queries/schema/search tools in-process.
    • Direct GraphQL for the deterministic pipeline reads and the upsertDataContract writeback (with SDK aspect-emission fallback stamping PROPOSED provenance) — the paths the Kit doesn't expose.
    • MCP Server (mcp-server-datahub): the judge-facing interactive surface. .mcp.json ships preconfigured — point Claude/Cursor at the demo catalog and interrogate the same lineage and PROPOSED contract the guardian used.
    • Ingestion (dbt + business-glossary sources) seeds the demo reality: dbt tests become assertions backing the contract; glossary versions are the semantic-drift baseline.
  • Narrative: Google ADK Agent, first-party DataHub tools, provider-agnostic by repo configuration: gemini-* runs ADK-native; any other id routes through LiteLLM against an OpenAI-compatible endpoint. The demo makes a real LLM api call per run (qwen3.6-flash ; see README "Choosing the narrative LLM"), and the comment attributes the narrative to the model that wrote it. A configured-but-keyless setup fails the check with the exact secret to add; only keyless forks fall back to a template summary, labeled as such. The LLM never scores either way.
  • Demo world: a fiction-retail dbt project (seeds → staging → fct_orders → revenue_daily) on BigQuery, with a formal business glossary attached via dbt meta.
  • Change detection reuses dbt's own state:modified semantics via manifest diff (dbt parse, no warehouse creds needed in CI), not a reinvented differ.

Challenges we ran into

  • DataHub's contract-proposal inbox is Cloud-only. I verified mid-design that proposeDataContract doesn't exist on self-hosted OSS. Rather than fake it, I upsert the contract with an explicit PROPOSED status aspect and made PR merge the approval gate — the human approval moved to where humans already are.
  • Assertions live on the wrong URN. Live verification against a real OSS instance showed dbt-test assertions attach to the dbt sibling URN, not the warehouse dataset URN. The client now merges both siblings before contract assembly.
  • upsertDataContract rejects unknown keys. My provenance fields had to move out of the upsert input into a separate status aspect emitted via the SDK.
  • dbt docs generate silently overwrites run_results.json — which destroyed test results before ingestion. Run order now: tests last, then ingest.
  • Empty env vars are not absent env vars. The composite action exported empty-string warehouse inputs, which silently defeated profiles.yml defaults. Fixed by only exporting non-empty inputs and env-indirecting everything (no direct ${{ }} interpolation in run scripts — also an injection-surface fix).

Accomplishments that we're proud of

  • The demo asymmetry works end-to-end: the staged breaking PR (column rename + silent metric redefinition + glossary drift) passes repo-internal CI green, and the guardian catches all three from metadata alone. That asymmetry is the pitch.
  • Full live-loop verification against a real self-hosted DataHub + BigQuery: lineage traversal, sibling dedupe, glossary drift, upsertDataContract + PENDING status aspect all confirmed working, contract inspectable via OpenAPI.
  • 48/48 deterministic tests passing in a fraction of a second — severity scoring and codegen are fully unit-tested precisely because no LLM touches them.
  • Offline mode is first-class, not a degraded afterthought: a fork PR with zero secrets still renders the complete impact comment from committed fixtures.
  • Generated code is mergeable as-is: real compat/legacy views + schema.yml tests in examples/generated/, produced from an actual run — with a requires_human flag for the cases codegen honestly can't map.
  • ~2.3 min end-to-end, PR opened → report posted (median of 21 successful demo runs; range 52 s–3.6 min). That covers checkout, dbt parse, the DataHub reads, the LLM narrative, both contract upserts and the comment — on a free GitHub-hosted runner.

What we learned

  • The metadata graph is the only honest source of blast radius. I went in planning to be clever about detection; the durable insight is that detection is easy and consequence is the hard part — and consequence is precisely what lineage + observed queries give you and nothing else does.
  • "Reality vs hypothesis" is a design principle, not a slogan. The moment you're tempted to ingest a PR branch's state into the catalog "just for the demo," you've built a machine that pollutes its own source of truth. Keeping DataHub reality-only simplified every downstream decision.

What's next for Downstream Impact Guardian

  • Prod-manifest refresh as a main-branch workflow — it's a script today; automating it is the obvious next step and the one thing a real adopter would ask for first.
  • Query-usage ingestion in the live demo so listQueries returns observed reads live (fixtures carry that part of the story today).
  • Richer rename detection — today's heuristic handles the 1-removed/1-added case and honestly flags the rest for a human; column-level lineage could close the gap.
  • Derived column-level lineage — the middle rung of the precision ladder. sqlglot per-column expression attribution and consumer-declared depends_on_columns (the SAFE verdict) already ship; the remaining rung is reading DataHub's column-level (schemaField) lineage so per-consumer impact becomes fact even when a consumer hasn't declared its dependencies. That closes the gap between declared and worst-case automatically — but it is not just a read: I measured my instance and the cross-model column edges aren't emitted in the first place (see the Feedback Prize answer), so this rung needs the ingestion side solved before the client side is worth writing.
  • Incident memory as a temporal knowledge graph. DataHub's aspects are event-sourced; layering a Graphiti-style bi-temporal graph over the guardian's findings would let the agent remember incidents across PRs — "this table's contract has been broken three times, twice by the same upstream job" — turning per-PR judgment into longitudinal judgment.

Built With

  • bigquery
  • datahub
  • datahub-agent-context
  • dbt
  • gemini
  • github-actions
  • google-adk
  • graphql
  • mcp
  • openai
  • python
  • slack
  • vercel
Share this project:

Updates