CASCADE — autonomous data-incident response on DataHub

When a check fails, CASCADE reads DataHub's lineage graph via MCP to find the root cause, map the blast radius, file a native incident, and alert the owner — autonomously, for $0.13 per investigation.

Comment

Data breaks at 2 a.m. CASCADE is the on-call engineer that never sleeps.

Every data team knows the page: a column silently breaks, and suddenly nobody knows which dashboards are wrong, what upstream change caused it, or who owns the fix. An engineer answers that by hand — reading lineage, clicking through catalogs, guessing — for hours, while wrong numbers reach executives.

CASCADE answers it in ~5 minutes, for $0.13, autonomously.

The thesis, measured — not asserted

DataHub's premise: it's a context problem, not a model problem. We turned that into a number. The same model, the same three incidents, with and without DataHub's graph:

  • Without DataHub (symptom only): 0/3 root causes — plausible, generic ETL guesses.
  • With DataHub (lineage via MCP): 3/3 — the correct upstream asset, annotated and acted on.

Graded structurally: a scenario counts only if the agent annotates the ground-truth root-cause URN — never for merely mentioning a table name. Single recorded run per scenario; scripts/regrade.py re-scores the shipped traces so judges can verify offline.

What it does

Hand CASCADE a failing check and it runs the entire incident response, end to end:

  1. Triage — reads the schema, confirms the affected column (type, NOT NULL contract, PII tag).
  2. Root cause — traces column-level lineage upstream to the exact asset responsible, with evidence, in 1–2 hops.
  3. Blast radius — maps every downstream dashboard, chart, and dataset now showing wrong numbers (35 in our demo incident).
  4. Acts — writes back into DataHub — raises a native DataHub incident (Incidents API), registers a guard assertion for the failing column, and annotates the root-cause asset.
  5. Routes — reads ownership from the graph and drafts a ready-to-send alert: cause, impact, fix.

How it uses DataHub (read AND write)

  • Reads via the DataHub MCP Server: get_lineage (column-level — the core reasoning substrate), list_schema_fields, get_owners, get_entities, search.
  • Writes back via native APIs: raiseIncident, upsertCustomAssertion, description updates. The MCP server had no incident-write tools — we built them, and both are packaged for upstream contribution (PR links will land in the repo and this page).

It doesn't just query the graph — it extends it, closing the read→reason→write loop.

Security: the catalog is untrusted input

Agents that read shared metadata can be prompt-injected through the catalog itself. CASCADE's shipped procedure treats every description as untrusted: when we planted a fake "incident note" pointing at a bogus root cause, it flagged the note and verified via get_entities before acting. Reproducible from a clean clone: python scripts/plant_injection.py.

Built to be trusted, not just demoed

  • 3/3 vs 0/3 blind eval, URN-graded, re-scoreable offline.
  • $0.13 per investigation — cheaper than one minute of engineer time.
  • Observe → propose → act rollout: --propose mode prints intended writes without executing; writes are allowlisted, budget-capped, and audit-logged (audit/audit.jsonl).
  • Webhook-ready: POST /api/trigger accepts an assertion-failure payload — the hook DataHub's Actions/webhook events would point at.
  • Tested + CI: 44 unit tests (grading, GraphQL client, API) running in GitHub Actions; pinned lockfile for reproducibility.
  • A real product — the "incident war room" web app replays three complete investigations from cached traces ($0 to judge) plus a gated live mode.

How we built it

Python + the Claude Agent SDK orchestrating the official DataHub MCP Server against a local DataHub quickstart (showcase-ecommerce: 1,049 entities with lineage). In-process MCP tools handle the write-back (GraphQL raiseIncident / upsertCustomAssertion) with reliable URN capture. FastAPI + a hand-built frontend for the war room. Model-agnostic by construction — any function-calling model works via MCP.

Challenges

Reliable URN capture from agent tool calls (solved with in-process tool state), GraphQL enum quirks, oversized lineage payloads vs agent context limits, and making an autonomous agent trustworthy — which shaped the propose mode, guardrails, audit log, and the injection defense.

What's next

DataHub Actions-framework consumer so assertion failures trigger CASCADE natively; reportAssertionResult run events so the guard assertion evaluates in the UI; FIELD-typed incidents with assigneeUrns; agent-driven incident closure once the guard passes.

Code: github.com/Biniyoyo/Cascade (Apache-2.0) — agent, war room, eval + regrader, tests/CI, and both OSS contributions.

Built With

  • anthropic-claude
  • claude-agent-sdk
  • datahub
  • docker
  • fastapi
  • graphql
  • mcp
  • python
Share this project:

Updates