Lineage Marshal

Inspiration

Data pipeline incidents are exhausting in the same way every time: something breaks, and the first hour is spent just figuring out what actually broke and who's affected — chasing lineage manually, pinging people who may not even own the asset, and starting from zero even if this exact thing happened last month. DataHub already has all of this context sitting in its metadata graph. We wanted an agent that actually uses it, not just reads it.

What it does

Lineage Marshal is an incident investigator agent for data pipelines. When something breaks — a schema change, a missed freshness SLA, a failed job — it:

Receives a trigger (simulated for the hackathon, designed to hook into Airflow/dbt/Kafka in production)
Reads DataHub via MCP — lineage, ownership, glossary/business context
Computes real blast radius — traverses downstream lineage and scores impact using usage signals, hop distance, and ownership risk, not just a raw count of downstream tables
Resolves the right owner with actual contact info, explicitly surfacing "no owner found" as a finding rather than failing silently
Generates a plain-English investigation report that calls out unknowns instead of hiding gaps
Writes the findings back to DataHub as a versioned Context Document on the affected asset — so the next investigation, by agent or human, inherits this knowledge instead of starting from zero

That last step is the differentiator. Most agents that connect to a metadata catalog are read-only — they answer questions about your data. Lineage Marshal contributes back to the graph.

How we built it

DataHub OSS, self-hosted locally, seeded with a demo dataset containing deliberately planted issues (broken lineage, missing ownership, stale freshness) so the agent has real problems to investigate
DataHub MCP Server (mcp-server-datahub), connected via a raw stdio JSON-RPC client — we discovered the real tool surface first before building anything on top of it
Typed Python wrappers around the MCP tools (search, lineage, ownership, glossary, usage stats, context document read/write), with graceful handling of missing data throughout
A trigger/detection layer with deduplication, a blast-radius scoring engine, ownership/glossary resolution, and a report generator — all chained into one investigation pipeline
Context document write-back using DataHub's InstitutionalMemory aspect, so each new investigation appends a new versioned entry instead of overwriting prior history
A FastAPI backend orchestrating the full pipeline, and a React frontend visualizing the investigation as it happens

Challenges we ran into

Self-hosted DataHub OSS uses MCP over a stdio subprocess, not an HTTP endpoint like DataHub Cloud — getting the connectivity model right took real investigation before writing any wrapper code
CLI/server version mismatches between acryl-datahub and our running GMS instance caused silent ingestion failures until we pinned versions
Getting write-back to actually version instead of overwrite required moving off a simple description-field mutation and onto the InstitutionalMemory aspect
Demo data didn't originally have any downstream lineage fan-out to actually demonstrate blast-radius ranking — we had to extend our seed data to give the scoring engine something real to rank

What we learned

That "does it write back" is a genuinely different bar than "does it read well" — most of the engineering effort was in making the write-back safe (versioned, non-destructive, failure-visible) rather than in the read/analysis side, which is often where agent demos stop.