LineageShield

Every downstream consequence, before merge.

Comment

Inspiration

A schema diff describes a local change, but its real risk lives in the metadata graph. A renamed feature may break an online model several hops downstream; a type change may affect a Tier 1 dashboard; a privacy transformation may alter an activation audience. Teams need those consequences before merge, not after an incident. LineageShield converts DataHub lineage, schema, ownership, tag, and tier metadata into an actionable change-review plan.

What it does

LineageShield analyzes a proposed schema or semantic change and produces:

  • a complete downstream blast radius across datasets, dashboards, charts, ML models, and activations;
  • a transparent risk score for every affected asset;
  • the accountable owners and reviewers for the change;
  • compatibility and policy findings;
  • generated compatibility SQL where appropriate; and
  • a phased rollout checklist that can be copied directly into a pull request.

The hosted demo includes three reproducible scenarios: hashing a plaintext customer email, narrowing a revenue decimal type, and renaming an online ML feature.

How we built it

LineageShield's primary integration is a read-only client for the official DataHub MCP Server over Streamable HTTP. It executes a four-tool context pipeline: search resolves the changed dataset, get_entities fetches ownership, tags, tiers, and properties, list_schema_fields narrows the affected contract, and get_lineage traces the downstream blast radius across three hops. Every tool name, argument, and response is preserved in a public, reproducible context trace while the bearer token is never written to disk. A direct acryl-datahub SDK snapshot exporter remains available for Core deployments that have not enabled MCP.

The TypeScript engine builds a typed adjacency index, traverses downstream lineage, and ranks impact using the change kind, criticality and tier, policy-sensitive tags such as PII and SOX, lineage depth, downstream count, and asset semantics. Recommendations remain deterministic and explainable: every risk and reviewer is traceable to metadata in the graph.

For performance, the traversal hot path uses a compressed sparse row index backed by Uint32Array offsets and targets. A native GitHub ARM64 workflow verifies result parity and enforces a minimum 2× speedup against an edge-scan baseline. The validated run measured a 1,963.94× median speedup on a 3,000-node, 8,997-edge graph while returning the same 2,819 reachable nodes.

Challenges we ran into

The main challenge was turning heterogeneous catalog metadata into decisions without hiding the reasoning inside an opaque score. We kept the graph format and traversal auditable, attached each finding to the metadata that triggered it, and separated the read-only DataHub MCP client from the hosted demo so no credentials are ever requested by the public app. We also normalized the MCP tools' different response envelopes into one evidence trace and added a deterministic fixture so the exact pipeline is testable without a tenant token.

Another challenge was keeping graph analysis responsive on large lineage networks. Replacing repeated edge scans with a typed CSR index preserved exact traversal results while dramatically improving locality and runtime on ARM64.

Accomplishments that we're proud of

  • Built a working end-to-end DataHub context-graph agent rather than a static catalog mockup.
  • Added a tested four-call integration with the official DataHub MCP Server, plus a public call trace and generated artifact examples.
  • Kept a read-only SDK fallback compatible with DataHub Cloud and current DataHub Core Graph APIs.
  • Made every risk, owner, policy finding, and rollout step explainable.
  • Generated merge-ready compatibility SQL and rollout checklists.
  • Shipped a hosted interactive demo, public source, tagged release, video, automated tests, and native ARM64 benchmark validation.
  • Verified a 1,963.94× median traversal speedup with exact result parity.

What we learned

Metadata becomes most valuable when it is converted into a concrete decision at the point of change. Lineage alone is not enough: ownership, tiers, tags, schema semantics, and downstream asset types must be evaluated together. We also learned that deterministic graph analysis can provide agent-like recommendations while remaining fast, reproducible, and easy for reviewers to audit.

What's next for LineageShield

  • Accept proposed schema diffs directly from CI and pull requests.
  • Add DataHub Actions integration so high-risk changes automatically request the correct owners.
  • Persist approval history and rollout evidence back into DataHub.
  • Support organization-specific risk policies and change templates.
  • Extend compatibility generation to additional warehouses and stream-processing systems.

Links

Built With

  • datahub
  • datahub-mcp
Share this project:

Updates