Second Responder

Inspiration

Every data team solves the same incident twice.

Root-cause analysis on a broken pipeline is expensive work — walking lineage, reading schemas, checking deploy notes, chasing the one column that quietly disappeared. And when it's over, the knowledge goes into a Slack thread and dies there. Six weeks later the same failure hits a neighbouring pipeline, and whoever is on call starts from zero.

We wanted to fix where that knowledge lives. DataHub already holds the map of what's connected to what. If an investigation writes its findings back into that map, then the map itself gets smarter — and the next responder, human or agent, inherits everything.

What it does

Second Responder is a three-agent on-call crew that investigates data incidents through DataHub and writes what it learns back into the graph.

• Triage always checks institutional memory first — searching DataHub Documents for prior postmortems before touching anything else — then scopes the blast radius.
• RCA pinpoints the failing asset using column-level lineage, exact transformation paths, and schema drift: the tell is a fine-grained lineage mapping that still points at a column the upstream table no longer has.
• Scribe writes the result back: a structured postmortem Document on the culprit asset, known-flaky-source / incident-history tags, a known-issue note on the broken BI asset, and structured properties recording incident count, date, and a link to the postmortem.

Each postmortem ends with a "Next time" recipe — literal, numbered verification steps naming the shared intermediate asset, written for whoever hits this pattern next.

Then the loop closes. When a lookalike incident hits a different pipeline, Triage's first tool call retrieves the crew's own postmortem, recognises the pattern, and RCA verifies the analogous culprit instead of re-deriving it.

The result we measured

Two incidents, same code, same prompts, only the graph state differs:

Both runs found the correct root cause. The complete event streams — every tool call, in order, with inputs — are committed under examples/ so you can diff them yourself rather than take our word for it.

How we built it

The crew runs on the Claude Agent SDK. Each agent is a fresh session with a role-specific prompt and no tools except the DataHub MCP server — no filesystem, no shell. Everything they know about the platform comes from the graph, and everything they learn goes back to it.

Reads: search, get_entities, get_lineage (column-level), get_lineage_paths_between, list_schema_fields, search_documents, grep_documents. Writes: save_document, add_tags, update_description, add_structured_properties.

Around that sits an incident injector that breaks a pipeline in the showcase-ecommerce datapack (a schema migration drops a column that downstream fine-grained lineage still references), a zero-dependency live console that streams every tool call with a running counter, and a replay mode that re-runs a committed event stream at any speed for demos.

Challenges we ran into

The honest one: our first version got 26 calls down to 21, which proves nothing. Memory that merely exists isn't memory that gets used. Two changes made the loop real — telling Triage to stop investigating on a memory hit rather than politely double-checking, and having Scribe write an explicit "Next time" recipe instead of a narrative postmortem. Prose describes what happened; a recipe tells the next agent what to do.

We also separated the counter into investigation calls versus write-backs, because counting the Scribe's writes as "search cost" muddied the only number that matters.

What we learned

Write-back is only worth points if something later reads it. The interesting design surface isn't storing knowledge — it's making retrieval deterministic (rigid document titles, a standing memory-first procedure) and making the retrieved artifact actionable enough that an agent will trust it over starting fresh.

What's next

Wiring alert intake to real monitors and assertions, raising native DataHub incident entities alongside the postmortem, and routing risky writes through DataHub's proposal workflow so a human approves before the graph changes.

Honest limits

The incidents here are injected metadata scenarios — the demo datapack carries no live data, so the break is a real schema change rather than a real query failure. The lineage traversal, the retrieval, the reasoning, and every write-back are genuine and reproducible on a laptop in minutes.

Built With

• anthropic
• claude
• datahub
• docker
• mcp
• python