DataHub Incident Memory Agent

What it does

DataHub already knows what connects to what (lineage) and what broke (assertions, data-quality checks). What it does not have is memory of how past incidents got diagnosed, so every on-call rotation re-derives the same triage from scratch.

This agent adds a cross-incident memory layer on top of DataHub:

• A data-quality event fires on an entity (e.g. a table lost ~90% of its rows overnight).
• The agent walks the real lineage through DataHub's get_lineage MCP tool and surfaces the nearest upstream dataset as a likely-cause candidate to triage first, the way a human on-call engineer would. (This is a triage shortcut, not a proof of causation.)
• It writes that candidate diagnosis back onto the entity with the update_description MCP mutation, so the next engineer sees it right in the catalog.
• It records the incident in its own memory (a hybrid FTS5 + embedding search over past incidents).
• When the same failure recurs, it recognizes it instantly from memory (match score 1.0) and returns the known diagnosis with no lineage walk and no LLM call.

Triage time goes from minutes of re-investigation to seconds. The novel part is the memory layer: DataHub provides the lineage and the writeback; this agent adds cross-incident recall that DataHub does not ship today.

How we built it

• DataHub via MCP, not REST. The agent reaches DataHub only through mcp-server-datahub (spawned over stdio), the same tool surface an interactive agent uses. A persistent MCP session (a background asyncio loop owns the stdio session for its whole lifetime) exposes plain sync methods to the pipeline. The main flow uses get_lineage and update_description; search and save_document are implemented and live-tested client methods for future extensions.
• Incident memory. A small SQLite store (incidents + incident_edges, graph-shaped) with find_similar_incidents() = FTS5 keyword retrieval + Jaccard token-overlap ranking, auto-upgrading to sentence-transformers embedding cosine similarity if that package is installed.
• Real instance, Docker-less dev. DataHub ran via datahub docker quickstart (v1.5.0.6) seeded with the sample metadata graph, inside a GitHub Codespace since the dev machine has no Docker. The lightweight uvx MCP server (pinned mcp-server-datahub@0.6.0) points at it. The whole loop was verified live and the writeback confirmed independently via GraphQL.

Challenges we ran into

• Bridging an async MCP session to a sync pipeline without hitting anyio's "cancel scope exited in a different task" rule, solved with one long-lived manager coroutine that owns the session and services calls off a queue.
• The MCP tools' real argument names differ from the obvious guesses (get_lineage uses upstream + max_hops, not direction; update_description uses entity_urn; save_document needs a document_type from a fixed set) — found by probing the live server.
• A mutation that silently "succeeded" without writing: the client now checks CallToolResult.isError instead of swallowing failed calls.

Accomplishments that we're proud of

The full loop runs against a real DataHub instance, and the payoff is visible: the same incident goes from a full lineage investigation (cold) to an instant memory hit (score 1.0) on recurrence, with the candidate diagnosis written back into DataHub itself.

What we learned

DataHub's MCP server is a clean, real substrate for agents: reading lineage and writing conclusions back with the same tool surface. The interesting layer to add is memory, the thing a catalog does not keep but an on-call engineer wishes it did.

Known limitations (honest scope)

• Cause selection is a triage heuristic (nearest upstream), not causal proof.
• Memory recall matches on error text only; it is not yet scoped by URN/lineage, so an unrelated asset with a similarly worded error could be recalled.
• Incident-graph edges (link_incidents) are not auto-wired yet; tests are module smoke tests. Details in the repo README.

What's next

• Scope memory recall by lineage/URN; richer cause signals (schema diff / assertions / logs); auto-link incidents; and (stretch) expose this memory layer as its own MCP server so other DataHub agents can query it.

Built With

• datahub
• docker
• github-codespaces
• mcp-server-datahub
• model-context-protocol
• python
• sqlite