InferGuard DataOps

A multi-agent DataOps system that uses DataHub context and lineage to detect failures, identify root causes, recommend fixes, verify recovery, and generate incident reports.

Comment

Inspiration

Data incidents rarely arrive as a single clear alert. A breaking schema change may first appear as an Airflow failure, then a dbt build error, a freshness violation, missing features, and finally a model or dashboard degradation.

Operations teams must manually connect these signals, search for the affected datasets, inspect lineage, identify the responsible upstream change, estimate the blast radius, and decide whether a remediation is safe.

We built InferGuard DataOps to explore a different approach: use DataHub as the trusted context graph for an agentic incident-response system that can move from alerts to root cause, controlled remediation, recovery verification, and postmortem generation.

What it does

InferGuard DataOps implements a multi-agent incident-response loop:

  1. Incident Commander creates and coordinates the incident.
  2. Signal Analyst aggregates alerts from Airflow, dbt, data-quality systems, feature stores, and model monitoring.
  3. Data Context Analyst calls DataHub through the official MCP server.
  4. RCA Investigator combines operational evidence with metadata and lineage to rank possible root causes.
  5. Safety Governor checks whether a remediation is allowlisted, low-risk, and reversible.
  6. Remediation Executor performs the approved action in a controlled demo environment.
  7. Recovery & Learning Agent verifies SLO recovery and generates a postmortem and reusable runbook.

The demo models a breaking change in raw.raw_orders: customer_id was renamed to customer_key, while the downstream staging.stg_orders transformation continued to reference the old field.

This causes failures in analytics.customer_ltv_features and affects downstream consumers, including a Feast feature table and the Retention Health dashboard.

InferGuard uses DataHub MCP to:

  • Search for the affected production asset.
  • Retrieve its schema and metadata.
  • Trace upstream lineage to identify the changed producer.
  • Trace downstream lineage to calculate the blast radius.

It then identifies the schema change as the most likely root cause, authorizes a reversible rollback in the sandbox, verifies freshness and availability SLOs, and generates an auditable postmortem.

How we built it

InferGuard is implemented in Python as a set of role-specific agents coordinated by an incident commander.

We deployed DataHub Core locally with Docker and created a demo metadata graph containing:

  • Snowflake source and transformation datasets.
  • A production customer feature dataset.
  • A Feast feature table.
  • A Looker retention dashboard.
  • An MLflow churn model.

The live integration launches the official open-source DataHub MCP server and uses the following read-only tools:

  • search
  • get_entities
  • get_lineage for upstream analysis
  • get_lineage for downstream impact analysis

Every observation receives an evidence ID and is stored in the final incident record. DataHub MCP results are explicitly marked as mode: live, making it possible to distinguish real metadata queries from deterministic offline fixtures.

The safety layer separates context gathering from remediation authority. DataHub access remains read-only, while remediation requires an allowlisted and reversible action. Recovery is evaluated independently against explicit SLO targets.

The final outputs include:

  • incident.json containing alerts, evidence, hypotheses, decisions, remediation, verification, and agent trace events.
  • postmortem.md containing the root cause, supporting evidence, remediation, recovery result, preventive action, and reusable runbook.

Challenges we ran into

One challenge was making an agentic workflow verifiable rather than producing only a natural-language recommendation. We addressed this by preserving evidence IDs, tool arguments, tool results, safety decisions, and actor-level trace events.

Another challenge was supporting different DataHub MCP tool schemas. The live adapter inspects the schemas exposed by the connected MCP server and only sends supported arguments.

We also needed to design a meaningful safety boundary. Metadata context can support a remediation decision, but it should not automatically grant permission to mutate production systems. InferGuard therefore keeps DataHub MCP access read-only and gives the Safety Governor final control over execution.

Finally, we implemented both live and fixture modes. Live mode demonstrates the real DataHub integration, while fixture mode provides a deterministic fallback for testing and offline evaluation.

Accomplishments that we're proud of

  • Successfully deployed a self-hosted DataHub environment.
  • Built and queried a multi-hop metadata and lineage graph.
  • Completed real DataHub MCP calls in live mode.
  • Connected alert aggregation, root-cause analysis, safety checks, remediation, recovery verification, and learning in one closed loop.
  • Produced auditable incident and postmortem artifacts.
  • Added deterministic tests for alert deduplication, safety rejection, rollback behavior, DataHub evidence collection, and end-to-end recovery.

In the live demo, InferGuard produces four verified DataHub MCP evidence records, identifies the schema regression with 82% confidence, passes all SLO checks, and closes the incident as recovered.

What we learned

We learned that DataHub can serve as more than a catalog used during manual investigation. Its metadata and lineage can become runtime context for operational agents.

Operational alerts explain what is failing, while DataHub helps explain what the asset is, what changed upstream, who depends on it, and how large the impact may be.

We also learned that safe automation requires more than root-cause confidence. Evidence provenance, reversible actions, explicit policies, independent recovery verification, and complete traces are essential for trustworthy agentic operations.

What's next for InferGuard DataOps

Next, we plan to:

  • Connect real Airflow, dbt, data-quality, and feature-store events.
  • Add Slack and ticketing-based human approval for medium-risk actions.
  • Execute real dbt rollback and replay workflows in an isolated environment.
  • Write incident status and postmortem links back into DataHub.
  • Add ownership, domain, data-contract, and assertion context to RCA.
  • Implement a lineage-aware compatibility gate that blocks breaking schema changes before deployment.
  • Evaluate the system against a larger collection of reproducible data incidents.

Our long-term goal is to turn DataHub's context graph into the foundation of a safe, explainable, and continuously improving data-operations control plane.

Built With

Share this project:

Updates

Submission history