LineagePulse

Multi-agent incident response for DataHub. Detects failing assertions, walks the lineage blast radius, writes a structured incident back to the graph, and notifies owners in one Slack message.

Comment

Inspiration

When a dataset breaks in production, data engineers spend 90 minutes clicking through DataHub, copying owner emails, and pinging Slack channels. The failure is usually obvious in retrospect and the impact is usually obvious to whoever owns the lineage graph — but no agent is watching it. We built LineagePulse to be that agent.

What it does

LineagePulse is a multi-agent system that closes the full incident-response loop on top of DataHub's context graph.

Three cooperating sub-agents (Detector, Investigator, Responder) read lineage, ownership, quality signals, and ML metadata via the official datahub-agent-context SDK, walk the upstream and downstream blast radius, and post a Slack message to the right owners — all in under 60 seconds. The structured incident record is then written back to the DataHub graph as a Document, so the next agent that touches the same asset inherits the knowledge.

When the blast-radius walk finds an ML model in the downstream graph, the agent automatically bumps severity to CRITICAL. A broken dataset that feeds a production model is a different class of incident than a broken dashboard, and the agent treats it that way. DataHub's first-class mlModel entity makes this inference structural rather than heuristic.

How we built it

  • Detector — polls DataHub for failing assertions via get_dataset_assertions and search, produces a structured Incident object.
  • Investigator — walks lineage with get_lineage, attaches owner / domain / description / quality via get_entities, and asks an LLM to author a root-cause + fix report.
  • Responder — writes the incident Document back to DataHub using save_document and posts a per-owner Slack message with severity, blast radius, root cause, and a fix. Built on the same LangGraph-style orchestration pattern as DataHub's own open-source Analytics Agent. Apache 2.0, pip installable, 18 unit tests, full demo runbooks, and a deployed live demo.

Challenges we ran into

The hardest part was wrapping the official datahub-agent-context LangChain tools behind a stable interface that worked in both production and DRY_RUN mode (so judges could try it without spinning up a DataHub instance). We solved this with a thin abstraction that decouples the agent logic from the SDK, and a test suite with mocked tool surfaces that verifies the right tool is called with the right arguments.

The other challenge was ML lineage — making the agent understand that an mlModel in the downstream graph means the incident is production-critical. We solved this by making the severity bump a property of the lineage walk itself, not a separate heuristic.

Accomplishments that we're proud of

  • Closed the read-write loop on DataHub — the agent doesn't just read the context graph, it writes incident Documents back so the next agent inherits the knowledge. The hackathon rubric explicitly rewards this property.

  • ML-aware severity bump — when the lineage walk finds an ML model in the downstream graph, severity automatically bumps to CRITICAL. A broken dataset that feeds a production model is a different class of incident than a broken dashboard, and the agent treats it that way using DataHub's first-class mlModel entity.

  • Three cooperating sub-agents with separate tool surfaces — Detector, Investigator, and Responder are independently testable, follow the same LangGraph pattern as DataHub's own Analytics Agent, and exchange state through a shared Incident object.

  • Production-shaped out of a hackathon build — Apache 2.0, pip-installable, 18 unit tests, full demo runbooks, Dockerfile, CI workflow definition, a deployed live demo on Render, and a 49-second screencast that runs in any browser.

  • Zero required credentials to try — judges can run the demo and inspect the full Slack / DataHub Document output without spinning up a DataHub instance, an LLM key, or a Slack workspace.

What we learned

The DataHub Agent Context Kit is genuinely production-shaped. The save_document tool closes the read/write loop in a way most catalog tools don't. We came away convinced that the right way to build agents on top of a metadata platform is to treat the graph as a read-write knowledge substrate, not a passive index.

What's next for LineagePulse

  • Schema-change detection (column adds / drops / deprecations)
  • A GitHub PR responder that opens a fix-PR with the suggested dbt model
  • A PagerDuty integration for SEV-1 incidents
  • A small web dashboard showing the recent incident feed

Built With

  • anthropic-claude
  • apache-2.0
  • data-engineering
  • datahub
  • datahub-agent-context-kit
  • datahub-mcp-server
  • datahub-skills-registry
  • fastapi
  • github
  • incident-response
  • langchain
  • langgraph
  • line-age
  • ml-metadata
  • mlops
  • multi-agent
  • python
  • render
  • slack
Share this project:

Updates