Data Incident Response Agent (DIRA)

Autonomous data incident response - from assertion failure to root cause report in 45 seconds, not 4 hours.

Comment

About the Project

Inspiration

Data teams today operate in a reactive loop: an assertion fails, a Slack ping goes out, and an engineer spends the next few hours manually tracing lineage in the DataHub UI, clicking through upstream datasets, checking schema history, looking at freshness, and trying to answer one question: what actually broke upstream?

We've lived this. Mean Time To Resolution for data incidents is measured in hours, not because the investigation is intellectually hard, but because it's tedious, a deterministic graph traversal that a machine should be doing. The tools exist (DataHub has lineage, assertions, and an MCP server), but nobody has stitched them together into an autonomous pipeline that goes from alert → root cause → report without a human in the loop.

That's what we set out to build.

What We Learned

DataHub's MCP Server is the real deal. The Model Context Protocol server exposes get_lineage, get_entities, search, save_document, and add_tags as clean tool calls. Once we had the MCP client wired up, the agents could interact with DataHub the same way a human does in the UI, just programmatically and 100x faster.

Agent orchestration is harder than it looks. The naive assumption is "chain a few LLM calls and you're done." In practice, the LLM is the least interesting part. The hard work is:

  • Parsing the heterogeneous entity metadata that DataHub returns (MCP format vs. legacy aspect format)
  • Designing a confidence scoring system that doesn't overfit to one scenario
  • Handling the combinatorial explosion of upstream lineage (3 hops can produce dozens of nodes)
  • Deduplication, the same assertion can fire repeatedly, and you don't want 17 identical incidents

Heuristics + LLM > LLM alone. Our Tracer and Checker agents use deterministic heuristic scoring (failed assertions = +0.5, schema change = +0.3, etc.) as the backbone, with the LLM as an optional reasoning layer that adjusts confidence by $\pm 0.2$. This hybrid approach is more reliable than pure LLM reasoning and degrades gracefully when the LLM is unavailable.

How We Built It

The system is an event-driven multi-agent pipeline:

$$\text{Assertion Failure} \xrightarrow{\text{Kafka}} \text{Actions Plugin} \xrightarrow{\text{IncidentEvent}} \text{Coordinator} \xrightarrow{\text{dispatch}} \begin{cases} \text{Tracer} \ \text{Checker} \ \text{Notifier} \ \text{Reporter} \end{cases}$$

DataHub Actions Plugin - A custom action that listens to the MetadataChangeLog_Timeseries_v1 Kafka topic, filters for AssertionRunEvent entities with FAIL status, and emits an IncidentEvent to the Coordinator.

Coordinator - Orchestrates the pipeline. Dispatches to sub-agents in sequence (Tracer → Checker → Notifier → Reporter), aggregates results, and manages deduplication via a SQLite incident store with a configurable time window (default: 900 seconds).

Tracer Agent - Calls get_lineage with direction=UPSTREAM and max_hops=3, then evaluates each upstream node using get_entities. Confidence scoring is additive:

$$\text{confidence} = \sum_{i} w_i \cdot \mathbb{1}[\text{signal}_i]$$

where $w_i \in {0.5, 0.3, 0.2, 0.1, 0.05}$ for failed assertions, schema changes, freshness issues, recently created nodes, and missing lineage respectively. Weights are configurable via config/agent_config.yaml.

Checker Agent - Takes the Tracer's candidates and validates each one by pulling fresh metadata, searching for related documents, and optionally invoking an LLM (Claude/GPT-4o/Gemini) to assess whether the candidate plausibly explains the assertion failure. Returns confirmed, probable, or rejected.

Notifier Agent - Formats a Slack alert with the dataset name, assertion URN, top root cause candidates with confidence scores, and a deep link to the DataHub dataset page. Supports configurable alert routing by platform and severity.

Reporter Agent - Generates a markdown incident report (summary, root cause analysis, lineage path, recommended actions) and writes it back to DataHub as a document via save_document. Tags root cause datasets with incident-root-cause using parameterized GraphQL mutations.

Dashboard - A React + Vite frontend that polls the agent's FastAPI server for incidents, stats, and health. Provides a real-time view of all incidents with status, root causes, and resolution times.

DataHub Skills - We used the Agent Context Kit's Skills feature to augment agent system prompts with domain-specific guidance (datahub-lineage, datahub-quality, datahub-enrich), giving the LLM context about DataHub's metadata model without bloating the base prompt.

Challenges We Faced

1. Heterogeneous entity formats. DataHub's MCP server returns entities as flat dicts with direct fields (name, health, schemaMetadata), while the legacy GraphQL API returns aspect-based structures (aspects: [{name: "schemaMetadata", ...}]). We had to build a parser that handles both formats, eventually extracted into a shared entity_utils.py utility after we caught ourselves duplicating ~100 lines between the Tracer and Checker.

2. GraphQL query correctness. DataHub's GraphQL schema is strict. We hit validation errors on enum values (quoting EQUAL as a string instead of an enum), wrong field names (dataset vs. datasetUrn in DatasetAssertionInfo), and overly complex filter clauses that the schema rejected. Each fix required reading the DataHub schema source, not just trial-and-error.

3. Deduplication semantics. The same assertion can fire every few minutes. Without deduplication, the pipeline would generate dozens of identical incidents. We implemented a time-window dedup keyed on assertion_urn:dataset_urn in a SQLite store, but had to carefully choose the window (900s) to balance between suppressing noise and not missing genuinely new failures.

4. LLM optional degradation. Not every deployment has an API key. We designed the system so the LLM is an optional confidence adjustor ($\pm 0.2$), not the primary reasoning engine. When no LLM is configured, the pipeline runs on pure heuristics and still produces useful results, just without the natural-language reasoning in the validation output.

5. End-to-end testing without a live DataHub. We wrote 134 tests using mock MCP clients that simulate DataHub responses in both MCP and legacy formats. This let us validate the full pipeline (Coordinator → Tracer → Checker → Notifier → Reporter) without requiring a running DataHub instance, while still catching format-parsing bugs.

What's Next

  • Parallel agent execution - Tracer and Checker currently run sequentially; they could run concurrently for independent candidates
  • Incremental lineage caching - avoid re-fetching the same lineage graph on repeated incidents
  • Multi-assertion correlation - detect when multiple assertions fail on related datasets simultaneously
  • Slack thread replies - post follow-up findings as threaded replies to the original alert

Built With

Share this project:

Updates