ImmuneAgent

• 

  The pipeline halts itself after detecting an active incident on 'order_details'.

• 

  The high-severity incident, raised automatically by ImmuneAgent, visible in DataHub.

Inspiration

Data quality incidents rarely announce themselves. When an upstream table gets corrupted, traditional pipelines keep processing it blindly — by the time someone notices, bad data has already reached dashboards, reports, or downstream models. I wanted to explore whether an AI agent could reason about a table's real business importance — not just detect that something broke, but decide how urgent it actually is — using the context DataHub already provides.

What it does

ImmuneAgent watches for data corruption and reacts autonomously. When a source table is flagged as broken, it reads DataHub's lineage graph to find every downstream table affected, checks their governance tags (e.g. "Authoritative Source", "Most Queried") to judge business criticality, and — based on that reasoning — raises a real Incident directly in DataHub via its GraphQL API. Any downstream pipeline can then check for that incident before running, and halt itself automatically until the issue is resolved.

How I built it

Built entirely in Python against a local DataHub Quickstart deployment (Docker + WSL2). The agent uses the DataHub Python SDK (datahub.sdk) to read lineage and metadata, and DataHub's GraphQL API to raise incidents. A small simulated pipeline script demonstrates the halt behavior by querying active incidents before "processing" a table. To confirm compatibility with MCP-based agent workflows, the project also includes mcp_lineage.py, which performs the same downstream lineage lookup through the official DataHub MCP Server (mcp-server-datahub) using the Model Context Protocol.

The flow: orders (corrupted) → lineage lookup → order_details (downstream) → evaluate criticality using tags → raise Incident in DataHub → downstream pipeline halts itself.

Challenges I ran into

The biggest one was scope: DataHub already ships incident and lineage tooling out of the box, so the real challenge was making sure my agent adds genuine reasoning on top of those primitives — deciding severity from governance signals — rather than just wrapping an existing API call. I also hit a Windows-specific bug in the DataHub CLI that required moving development into WSL.

Accomplishments that I'm proud of

The full loop runs end-to-end against a real DataHub instance, no mocks: detection, reasoning, and a real Incident created and visible in the DataHub UI — which then genuinely halts a downstream process.

What I learned

How much of DataHub's platform is already built for exactly this kind of agentic workflow — lineage, incidents, and governance tags all compose naturally once you know they're there.

What's next for ImmuneAgent

Packaging the agent as a reusable DataHub Skill, adding support for multi-hop lineage, and generating human-readable incident explanations automatically.

Known Limitations

This demo uses the local DataHub Quickstart deployment. Authentication is intentionally disabled, since the focus of this project is the autonomous reasoning and incident-generation logic, not production deployment hardening.

Built With

• datahub
• docker
• graphql
• python
• wsl2