LineageGuard

Know the blast radius before you merge.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for LineageGuard

Inspiration

A schema change can look harmless in a pull request while silently breaking an executive dashboard, an Airflow job, and a production ML model. The evidence already exists in DataHub; teams need it converted into an accountable decision before merge.

What it does

LineageGuard is a schema-change safety agent. It gathers schema, ownership, tags, and multi-hop column lineage from DataHub, then returns an explainable ALLOW, REVIEW, or BLOCK decision. It identifies every impacted asset, names the specific risk, creates owner-scoped recovery tickets, and seals the decision in a deterministic SHA-256 receipt. A tested dual-write, backfill, contract-test, and rollback plan can move a blocked change into review.

How we built it

The Python agent is designed around DataHub's official MCP tools: get_entities, list_schema_fields, and get_lineage supply evidence, while add_tags and save_document are generated only as previewed, reversible actions. A deterministic policy engine keeps safety decisions reproducible and treats every catalog description as untrusted input. The browser demo runs from a bundled fixture so judges can test the full workflow instantly without credentials; the production adapter is ready for an operator-configured DataHub MCP server.

Challenges we ran into

We did not have an authenticated DataHub tenant, so we avoided pretending the hosted demo was live-connected. Instead, we built a defensive MCP adapter, a realistic Snowflake-to-dbt-to-Airflow-to-Looker/MLflow fixture, and explicit fail-closed behavior for uncertain provenance or missing ownership.

Accomplishments that we're proud of

LineageGuard ships as a working zero-install web demo plus a standard-library local server. Three focused tests prove that PII/model dependencies block unsafe changes, a complete migration plan produces review, and receipts are deterministic. The repository includes Apache-2.0 licensing, architecture notes, sample input, and a public 80-second demonstration.

What we learned

Metadata becomes operationally valuable when an agent preserves provenance, makes uncertainty visible, and turns context into reversible action rather than prose. DataHub's context graph is the right control plane for that job.

What's next

Connect an authorized live tenant, add versioned MCP response conformance fixtures, integrate LineageGuard as a CI merge check, and persist signed decision receipts back into DataHub after human approval.

Built With

Share this project:

Updates