Data Incident Crew

Four Claude agents that catch the silent data failures your catalog can't see — and write the whole investigation back into DataHub so the next person inherits it, not just the symptom.

Comment

Inspiration

Every data platform team knows this pattern: a pipeline stage stops writing real data, or a load "succeeds" with zero rows, and the catalog doesn't notice. Ingestion timestamps still update. Lineage still renders. Nobody gets paged. By the time a human notices, dashboards have been wrong for days. Metadata says fine — the data is lying.

We wanted to build an agent crew that does the runbook a human on-call engineer actually follows: confirm the failure by looking at the real data, figure out what's downstream, find the owner, write up what happened, and leave a trail so the next person doesn't re-derive it from scratch. And we wanted DataHub to be the substrate the crew writes to, not just a read-only source it summarizes.

What it does

Data Incident Crew is four specialized Claude agents — Detector, Blast-Radius, Root-Cause, and Scribe — that run in sequence against a real nyc-taxi SQLite pipeline (raw_trips → staging_trips → mart_daily_summary, ingested into DataHub with lineage):

  1. Detector queries the actual pipeline data (not catalog metadata) and finds a planted, catalog-invisible defect: staging_trips silently stops 9 days before its upstream table, plus a disclosed empty-load day in the mart.
  2. Blast-Radius walks DataHub's live lineage graph (never a hard-coded URN list) to compute exactly what's downstream and who owns it.
  3. Root-Cause builds a grounded RCA from the query history and the same evidence.
  4. Scribe writes every finding back into DataHub: an incident, a tag, a description note, a postmortem knowledge document, structured properties (io.incident_crew.severity / io.incident_crew.detected_at), and an assertion + result.

And the loop closes. Once an engineer fixes the underlying job, a deterministic incident-crew resolve command verifies the fix against the data, resolves the open incidents, flips the crew's assertions to SUCCESS, and removes the stale_data tags — live-verified end-to-end: the repair backfilled 39,640 staging rows, then resolve closed both incidents and cleaned both tags. The catalog round-trips instead of accumulating stale incident debris: detect → write back → fix → verify → resolve.

A --control mode runs the same crew against a clean instance and reports healthy with SUCCESS assertions — proving nothing is hard-coded to always find a problem.

How we built it

  • Claude Agent SDK (Python) orchestrates four ClaudeSDKClient sessions, one per agent role, each scoped to its own least-privilege tool allowlist. Sequencing, typed JSON contract validation (DetectorFindings → ImpactMap → RcaReport → WriteBackReceipt), and retry-once-on-failure are handled by our own Python orchestrator — not left to an LLM deciding when to hand off — so failures are deterministic and every stage failure leaves a forensic trail (runs/<ts>/events.jsonl).
  • DataHub MCP server (mcp-server-datahub==0.6.0, stdio, mutations enabled): search, get_entities, list_schema_fields, get_lineage, get_lineage_paths_between, get_dataset_queries for reads, and add_tags, update_description, add_structured_properties, save_document for catalog-enrichment write-back.
  • DataHub Python SDK / GraphQL, dual write path: the MCP server has no incident or assertion tools, so we built an in-process SDK MCP server wrapping DataHubGraph.execute_graphql (raiseIncident, updateIncidentStatus) and upsert_custom_assertion plus a hand-rolled reportAssertionResult mutation — both paths behind one module boundary.
  • Vendored DataHub Skills: the official datahub-lineage and datahub-quality skill instructions are injected verbatim into the Blast-Radius and Detector agent prompts.
  • Official nyc-taxi dataset (datahub-project/static-assets), pinned to a specific upstream commit, as both the incident instance and a clean control instance — no synthetic data for the core scenario.
  • A deterministic detection rule implemented in code (not LLM judgment) so the pass/fail boundary is unit-testable; 181 unit + 8 live integration tests; ruff clean; STRIDE security audit before release.

Challenges we ran into

  • GraphQL schema drift between SDK 1.6.x and the 1.5.x quickstart server, found by live-probing rather than trusting docs: updateIncidentStatus takes urn as a top-level argument (not nested in input), and the SDK's reportAssertionResult document references a type the server rejects even when null. We hand-rolled the exact mutation shapes the live server accepts.
  • A search-index race: the lineage-linking step discovers entities via search, which silently found zero entities right after ingestion. Fixed with an index-catch-up poll.
  • The first true end-to-end run found a critical bug the deterministic layer couldn't catch: the DataHub MCP server had never actually connected — its PyPI pin was actually the MCP protocol version (3.4.4), not a real release, and the CLI's MCP spawner doesn't merge our env (so bare uvx had no PATH). Fixing that surfaced four more real bugs, each found only by running the crew: an unscoped --instance write guard, a never-ingested control instance, an agent-controllable field_path that silently broke assertion reporting, and agent-supplied assertion timestamps that corrupted DataHub's health view in both time directions (we removed the parameter from the tool surface entirely). All were root-caused against the live server and fixed with regression tests.

Accomplishments that we're proud of

  • A real write-back trail landing in DataHub across two distinct write paths (MCP mutations + hand-corrected GraphQL), kept behind one clean module boundary.
  • The full incident lifecycle, live-verified: detect → write back → fix → verify → resolve — the catalog round-trips instead of accumulating stale incident debris.
  • Catching and fixing live GraphQL schema mismatches that no amount of reading SDK documentation would have surfaced — only probing the running server did.
  • A deterministic, unit-tested detection rule underneath a non-deterministic LLM crew, so the demo's pass/fail boundary never depends on how an agent phrases its findings.
  • A STRIDE security audit with zero CRITICAL findings, done before public release rather than after; every hazardous agent-controllable tool parameter removed from the tool surface.

What we learned

  • SDK documentation for a fast-moving GraphQL API is not a substitute for probing the actual running server version — the mutations that needed hand-rolling were exactly the ones the docs described incorrectly.
  • Typed, validated Python contracts between agents turn "the demo might say something different every run" into "the demo's structure is guaranteed every run, only the phrasing varies."
  • Least-privilege tool scoping per agent (no agent has both SQL read access and any write tool) is cheap to design up front and meaningfully shrinks the blast radius of a prompt-injected stage.
  • The bugs that matter most are only findable by running the full agent system against the live server — unit tests and even live tests of the deterministic layer all stayed green while the flagship path was broken.

What's next for Data Incident Crew

  • Slack/PagerDuty hand-off: notify a channel or on-call rotation the moment an incident is raised, instead of requiring someone to open the catalog.
  • Real warehouse adapters: swap the SQLite read-only adapter for a thin interface over Snowflake/BigQuery/Postgres so the same crew runs against production warehouses.
  • Scheduled sentinel mode: run the crew on a cron so silent failures get caught within an SLA window rather than whenever someone thinks to run it.

Built With

  • anthropic
  • claude
  • claude-agent-sdk
  • datahub
  • docker
  • github-actions
  • graphql
  • mcp
  • playwright
  • pytest
  • python
  • rich
  • ruff
  • sqlite
  • typer
  • uv
Share this project:

Updates