Lineage Relay

• 

  Lineage Relay returns NEEDS_OWNER rather than issuing a false-green schema change.

Inspiration

A column rename looks tiny in a migration ticket. In a real data stack it can quietly break a dashboard, an ML feature, or the relationship between a team and the data they own. The information exists in the catalog, but it is rarely assembled into one release decision a human can trust.

What it does

Lineage Relay reviews a proposed rename of orders.customer_id to buyer_id. It reads schema, ownership, and sensitivity metadata from DataHub, then uses the official DataHub MCP server to trace the exact field paths to analytics, a dashboard, and ML features.

It returns one explicit posture:

• NEEDS_OWNER when sensitive downstream use has no accountable owner.
• READY when ownership and compatibility actions are present.
• BLOCKED_BY_GOVERNANCE when a removal rule forbids the release.

For reviewable outcomes it generates a migration, compatibility view, contract test, and change summary. It never deploys them. Each decision writes a receipt and evidence hash back to the DataHub source asset.

Why it is different

Lineage Relay is not a chat answer that says a change seems safe. The release posture depends on observable metadata, exact MCP lineage paths, and accountability. The default demo is the missing-owner case because optimistic automation commonly misses that failure.

How we built it

A public-safe synthetic DataHub graph holds the source, analytics model, dashboard, ML feature, field lineage, ownership, and PII metadata. The app uses the DataHub Python SDK for schema and owner reads plus receipt write-back. The official DataHub MCP server provides get_lineage_paths_between traces that prove each source-to-consumer path at column level.

A deterministic decision engine selects the posture and review artifacts. FastAPI serves the review desk, which puts the field path, owner gap, decision, receipt, and generated package on one screen.

Challenges

The hard part was keeping the claim honest. Field lineage, ownership, and release posture had to be live evidence rather than hard-coded storytelling. We also kept MCP mutation tools disabled: they prove relationship paths, while the app's bounded DataHub SDK write records only the review receipt.

Accomplishments

• Proved all three release postures against a live synthetic DataHub graph.
• Verified three MCP field-level traces per review.
• Published the Apache-2.0 source, focused tests, reproducible lab configuration, and a 49.9-second public walkthrough.
• Made the unsafe shortcut visible: no owner means no false green light.

What we learned

A code-generation agent needs context before it generates. Field lineage and governance are useful only when they change the release decision and leave a durable handoff for the next reviewer.

What's next

The next useful expansion is not autonomous migration. It is a review queue for multiple proposed changes, with the same evidence-first contract and an explicit human approval boundary.

Built With

• datahub
• docker
• fastapi
• python