LineageGuard

Inspiration

A SQL migration can be syntactically valid and still break dashboards, machine-learning features, and data contracts. The missing context—schema, ownership, classifications, and downstream lineage—often already exists in DataHub, but is absent at the exact moment an engineer reviews the change.

What it does

LineageGuard is a DataHub-powered agent and merge gate for SQL schema changes. It:

• retrieves the target dataset, schema fields, and downstream lineage through the official DataHub MCP Server;
• detects dropped, renamed, type-changed, and newly required columns, plus destructive table, view, schema, and database operations;
• traces table- and column-level blast radius;
• raises severity for PII, sensitive data, glossary classifications, and missing ownership;
• returns an evidence-backed risk score and one of block, manual_review, or pass_with_checks;
• fails a GitHub check when the deterministic policy blocks a migration; and
• after explicit approval, can save the review decision back to DataHub as durable evidence.

The same analyzer powers the Agent Skill, CLI, composite GitHub Action, JSON and Markdown reports, API, and interactive console.

How we built it

The bundled DataHub Agent Skill calls the official MCP tools get_entities, list_schema_fields, and table- and column-level get_lineage. It gathers only the fields affected by the proposed SQL, paginates responses, and passes a compact evidence file into a deterministic Python policy engine.

SQLGlot parses dialect-aware SQL. Pydantic validates catalog evidence and verdicts. Typer provides the CLI, while FastAPI serves the local API and review console. A composite GitHub Action runs the exact same gate in CI and exits nonzero for a blocking verdict.

For repeatable judging, the repository includes a credential-free DataHub export fixture and committed expected Markdown/JSON output. Fixture mode is clearly separated from the live MCP connection.

Challenges

The hardest problem was making the system fail closed without creating noisy false positives. LineageGuard ignores comments and string literals, verifies that the migration actually targets the selected DataHub asset, distinguishes a newly created downstream view from an incorrect target, and requires manual review whenever DataHub returns incomplete schema or lineage evidence.

We also separated agentic catalog access from deterministic enforcement. The agent retrieves and can write evidence, while the merge decision remains reproducible locally and in CI.

Accomplishments

• Installable DataHub Agent Skill using the official universal MCP endpoint and OAuth-capable hosts
• Standalone tenant MCP client for token-based automation
• A reusable composite GitHub Action that blocks unsafe merges
• Table- and column-level lineage evidence
• Explicit approval gate before any DataHub writeback
• MySQL MODIFY and CHANGE COLUMN support alongside dialect-aware parsing
• 47 passing automated tests, lint/type checks, package build verification, and an independent audit with no remaining high-severity findings
• A real 66-second, 1080p product walkthrough

What we learned

Metadata is most valuable when it changes a concrete engineering decision. Ownership, classifications, and lineage are not merely catalog fields here: they determine severity, identify accountable people and affected systems, and shape the safe expand-and-contract rollout plan.

What's next

Next steps include pull-request annotations, configurable organization policies, usage-statistics signals, additional SQL dialect fixtures, and richer DataHub writeback that links the stored review to the affected asset and deployment.

Built With

• datahub
• datahub-mcp-server
• fastapi
• github-actions
• pydantic
• python
• remotion
• sqlglot
• typer