LineageTX

Inspiration

A producer schema rename can look tiny in a pull request while breaking models, orchestration code, and semantic contracts owned by different teams. Impact analysis alone tells us who may break; it does not make the change safe. We built LineageTX to coordinate that work before the producer change is allowed through.

LineageTX is inspired by two-phase commit, but it does not claim database-style atomicity across repositories or deployed systems.

What it does

The bounded scenario renames customer_id to customer_key. DataHub OSS v1.6.0 and the official DataHub MCP server discover the trusted schema, three-hop column lineage, owners, Tags, Structured Properties, and path evidence for exactly three consumers:

• A dbt SQL model receives a narrow automatic repair.
• An Airflow consumer requires a coordinated two-file mapping update.
• An ambiguous semantic mapping receives zero automatic writes and pauses for the exact DataHub owner to approve the exact mapping.

The coordinator moves through DETECTED → PREPARING → NEEDS_APPROVAL → PREPARED → COMMITTED, or ABORTED on failure. Candidate changes live only on isolated Git worktrees and branches. SQLGlot, AST and JSON checks, repository tests, expected SHAs, fixed file allowlists, lineage completeness, and owner policy decide whether a candidate is admissible.

COMMITTED requires three real candidate commit SHAs, an unmerged coordination receipt, a producer-gate release receipt, and 3/3 VERIFIED consumers. LineageTX never auto-merges. ABORT only cleans its own unmerged candidate changes; it does not claim to roll back deployed systems.

Why DataHub is essential

DataHub is the transaction's source of impact and governance truth. Column-level, multi-hop lineage defines the complete participant set. Schema, ownership, Tags, and Structured Properties bind each proposal to trusted context. Every migration state, owner, participant status, and evidence link is written back to DataHub and then read back and hashed. Without the data graph, LineageTX cannot know when preparation is complete or whom to ask for the unsafe semantic decision.

How we built it

The coordinator is implemented in Python 3.11 with DataHub OSS v1.6.0, the official MCP server, SQLGlot, pytest, and hardened Git worktrees. The successful path uses deterministic structured candidates; model output never authorizes itself. The public Vercel interface is explicitly labeled as a verified interactive replay with no browser credentials or external mutation. The under-three-minute video records the real DataHub OSS run and its evidence bundle.

Challenges

The hardest parts were proving complete discovery rather than accepting a partial graph, keeping candidate execution isolated from hooks and credentials, authenticating an exact owner approval without allowing proposals to self-authorize, and making DataHub write-back verifiable through read-back receipts.

Accomplishments and lessons

The final run reaches COMMITTED with three verified consumers and zero unverified consumers while leaving base repositories untouched and auto_merge=false. The evidence manifest seals context, proposals, diffs, verification, approval, write-back/read-back, candidate SHAs, and publication receipts with SHA-256.

We learned that a useful agent is not the component that generates the most edits. It is the component that knows the full blast radius, refuses ambiguous work, and produces evidence a human can audit.

What's next

The winning scope is intentionally frozen: one producer rename, three consumers, one approval boundary, and no automatic merge. The next practical step would be connecting the existing fixed-scope publisher to real organization repositories after repository-specific authorization and policy review.

Built With

• datahub
• datahub-mcp
• docker
• git
• github
• javascript
• pytest
• python
• sqlglot
• vercel