LineageFix

Inspiration

Schema changes are where data pipelines break quietly. Someone drops or retypes a column, the change looks small, it passes review, and three downstream dbt models fail in production hours later. The tools that react to this tend to swing between two extremes: they either auto-patch everything and hide the risk, or they block everything until people learn to ignore them. We wanted something in between. A reviewer that looks at real evidence before it says anything, and that stays silent when it does not know. One rule shaped the entire project: never fabricate evidence. If LineageFix cannot see lineage, a contract, or actual usage, it says so instead of guessing.

What it does

LineageFix takes a before and after schema for one dataset and checks it against three independent evidence sources:

1. Bounded downstream lineage from DataHub, capped at two hops. A truncated blast radius is treated as danger, never waved through.
2. The dbt source contract: declared column types and constraints, read from the checked-in project.
3. Real repository usage: which SQL models actually reference the changed column.

An arbiter compares the three and returns exactly one decision: safe, conflicted, needs_review, unsafe, or no_repair. Even safe means a human still reviews before merge. The tool only ever proposes constrained dbt repairs, like restoring a compatibility alias, and it never writes to a repository. Missing configuration returns not_ready. Missing evidence returns needs_review. It does not invent lineage results, validation output, commit SHAs, or pull request URLs.

How we built it

The backend is FastAPI with Pydantic, split into single-purpose modules that stay import-light. diff does deterministic schema classification with no I/O. collectors performs a read-only scan of a dbt project for contract and usage facts. adapters keeps all DataHub contact behind a small Protocol. planner runs the arbitration ladder. readiness probes environment configuration. Validation lives in the Pydantic models, so malformed input is rejected with a 422 before any adapter runs.

The frontend is a single-view Vite + React review workspace. It shows dependency readiness, the schema diff, bounded lineage impact, the evidence facts, and the final decision with its evidence trail.

DataHub runs as self-hosted open source via the official Docker quickstart. The backend talks to it through the acryl-datahub SDK with no token, because a local OSS instance runs with authentication disabled. That is the honest setup for anyone who cannot pay for a managed catalog.

Challenges we ran into

The hard part was designing a system that cannot lie to you. Absence of evidence had to become a first-class signal: a column with no declared type, an unreadable file, or a missing lineage source each produces no fact plus a note, which drives the arbiter to needs_review. We also had to get bias direction right. Textual usage counting can over-count a column name, so we made sure over-counting can only push a decision toward conflicted and never toward safe.

DataHub Cloud is a paid product and its free tier is aimed at startups, which we are not. We run the open-source edition self-hosted, and early on our own code demanded a cloud-style API token and refused to start without one. Making the token optional, which the SDK fully supports for auth-disabled OSS instances, was a small fix that changed who the tool is for.

We also hit the classic silent-failure trap: acryl-datahub does not support Python 3.9, and on 3.9 the dependency extra resolves to nothing while pip reports success. Lineage reads then fail at runtime instead of install time. We now require Python 3.10+ and document the failure mode. Finally, the quickstart stack is several gigabytes of container images, and watching that pull on a slow connection with a submission deadline is a challenge of its own.

Accomplishments that we're proud of

The arbitration ladder is the core. It checks decisions in a fixed order, and the order is the policy: unbounded impact is unsafe, missing evidence is needs_review, disagreeing sources are conflicted, and only a removed column with no remaining references and no contract obligation can reach safe with a repair proposal. Reordering the ladder changes behavior, and the tests pin it down.

We are proud of the honest degradation. Every missing credential, absent fact, and truncated query has an asserted, non-fabricated outcome. We are also proud of the evidence loop: POST /collect returns facts for a human to inspect, and the caller passes those exact facts back to POST /analyze, so the evidence a reviewer sees is byte-identical to the evidence the arbiter consumed. And we got the whole thing running end to end against a real local DataHub, with no cloud account and no token.

What we learned

Bias direction matters more than accuracy. A counter that sometimes over-counts is acceptable if the error only moves decisions toward refusal. Precise language keeps a design honest: "ready" means configured, not reachable, and writing that down stopped us from overclaiming. We also learned that a deliberately incomplete parser beats a speculative one. Every gap we refused to fill with a guess became a correct needs_review instead of a confident mistake. Finally, constraints improved the design. Having no budget for a managed catalog forced us to treat open-source DataHub as the first-class target, and the configuration boundary is cleaner for it.

What's next for LineageFix

Next we add the write path, still gated on evidence. Validation commands will come from the target repository's own configuration, never a universal assumption. A GitHub pull request opens only after that validation passes. The arbitration decision gets written back to DataHub so the catalog keeps the record. On the read side we plan deeper dbt analysis: ref() chain traversal, model-level contracts, and SQL parsing beyond word-boundary counting.

Built With

• python