LineageGuard

Inspiration

A schema change can look harmless in a pull request while silently breaking an executive dashboard, a customer-facing stream, or a production ML model several hops downstream. Traditional schema linters see the diff, but they do not know the organization's real lineage, ownership, criticality, or governance context.

LineageGuard closes that gap. It gives an AI coding agent or CI job the context needed to answer the question every reviewer actually cares about: what will this break if I merge it?

What it does

LineageGuard compares a proposed schema with the current data contract and uses DataHub downstream context to:

• Detect field drops, probable renames, incompatible type changes, tightened nullability, required additions, and lost sensitivity tags.
• Follow each change through field-aware lineage to datasets, dashboards, data jobs, streams, and ML models.
• Batch-enrich live lineage with DataHub ownership, tags, names, and platforms, while preserving the CLI's capped-result metadata.
• Score risk deterministically using asset criticality, governance tags, lineage depth, and field usage.
• Refuse to treat empty or partially enriched lineage as proof that a change is safe.
• Return a merge decision (ALLOW, REVIEW, or BLOCK) with the exact downstream URNs that justify it.
• Generate a compatibility SQL view, dbt schema tests, SARIF findings, and owner-specific notification drafts.
• Produce a polished zero-dependency HTML review dashboard plus Markdown and JSON reports.
• Plan createTag, batchAddTags, and updateDescription operations so the decision can be written back to DataHub.
• Keep every mutation behind an explicit approval gate; normal analysis is read-only.

How we built it

The dependency-free Python engine models proposed contracts as JSON and enriches them through a live DataHub CLI adapter. Live mode runs downstream discovery with skill attribution:

datahub -C skill=datahub-contract-guardian lineage \
  --urn '<DATASET_URN>' --direction downstream --hops 3 --format json

The adapter then follows DataHub's recommended batch-enrichment pattern: it collects the returned URNs and resolves up to 50 at a time through datahub search with a GraphQL projection for ownership, tags, names, and platforms. It records whether traversal was capped, distinguishes zero observed edges from verified absence, and degrades visibly if rich metadata cannot be retrieved.

The repository also contains an installable Agent Skill that composes DataHub's official lineage and enrichment patterns. It instructs agents to discover context first, present an exact write-back plan, and request human approval before mutations.

For a judge-friendly demo without credentials, the repository ships a deterministic showcase-ecommerce scenario with realistic DataHub URNs, field usage, owners, criticality, governance tags, and three-hop lineage. The same analyzer and report pipeline runs in fixture and live modes.

GitLab CI dogfoods LineageGuard on every commit, publishes all generated artifacts, and deploys the HTML dashboard to GitLab Pages.

Challenges we ran into

The hardest problem was turning heterogeneous metadata into a decision that is explainable and reproducible. A probabilistic answer would be difficult to trust in CI, so the core risk engine is deterministic and fully tested. We also had to normalize multiple DataHub CLI response shapes while preserving field-level context and distinguish a plan from an approved catalog mutation.

Another challenge was generating useful remediation instead of a generic warning. LineageGuard reconstructs the previous contract as a compatibility view, maps explicit rename hints, protects incompatible types with safe casts, and generates dbt assertions from the old contract.

Accomplishments we're proud of

• A complete end-to-end workflow from schema proposal to public review dashboard.
• Thirteen passing deterministic tests covering risky and safe changes, official DataHub CLI response shapes, 50-entity enrichment batching, context-completeness warnings, reports, generated code, a clean-checkout launch, and approval-gated write-back.
• Seven judge-visible outputs: HTML, Markdown, JSON, SARIF, SQL, dbt YAML, and owner notifications.
• A public Apache-2.0 repository with checked-in judge artifacts and a reproducible 1:57 narrated video.
• A reusable DataHub Agent Skill that can guide future agents through the same safe workflow.

What we learned

DataHub's value for agents goes beyond schema retrieval. Lineage, ownership, criticality, tags, and the ability to contribute context back to the graph turn a code generator into an impact-aware collaborator. We also learned that mutation safety is part of product quality: the best agent is not the one that writes fastest, but the one that shows exactly what it intends to change and waits for approval.

What's next

• Parse dbt manifests and SQL migration diffs directly, without requiring a hand-authored change specification.
• Add GitHub and GitLab merge-request comments with interactive approval controls.
• Use DataHub MCP tools as a second live provider alongside the CLI.
• Track deprecation windows and verify that downstream consumers have migrated before removing compatibility views.
• Write a structured decision document back into DataHub so later engineers and agents inherit the full rationale.

Built With

• css
• datahub
• datahub-cli
• datahub-skills
• dbt
• gitlab-ci/cd
• gitlab-pages
• graphql
• html
• python
• sarif
• sql