SchemaFlight

A blast-radius-aware compiler that turns risky schema renames into DataHub-grounded migration bundles a data team can review and merge.

Comment

Inspiration

A schema rename is rarely local. A developer may see one table and one column while the metadata graph sees downstream models, dashboards, exports, PII tags, observed queries, and owners. Code generators that miss that context produce migrations that are syntactically plausible and operationally dangerous.

SchemaFlight asks a practical question: can DataHub context turn a breaking rename into code a real data team would merge?

What it does

Given a rename_column request, SchemaFlight reads live schema, column lineage, ownership, field tags, and recorded query usage through the official DataHub Agent Context Kit. It fails closed when evidence is incomplete, rejects a direct rename when downstream or query usage exists, and emits a deterministic migration bundle:

  • expand, migrate, validate, contract, and rollback SQL;
  • SQL-AST-based downstream query patches with ambiguity routed to review;
  • a machine-readable impact manifest with lineage edges and owners;
  • a human migration decision and acknowledgement record; and
  • a self-contained evidence report.

The generated lifecycle is executed against DuckDB in tests. SchemaFlight never auto-executes production SQL. Write-back is a separate explicit flag that saves the decision as a DataHub document related to the source dataset.

How we built it

The compiler core depends on a small catalog protocol. The reproducible adapter reads a committed ecommerce snapshot; the live adapter wraps the official Agent Context Kit tools for entities, schema fields, lineage, and dataset queries. A deterministic breadth-first traversal preserves lineage depth and edges, while strict completeness checks reject truncated live evidence.

Migration SQL uses quoted identifiers and a controlled DuckDB type surface. SQLGlot rewrites only unqualified references in single-source statements; qualified or multi-source ambiguity remains untouched and is listed for human review. Generated artifacts are path-contained, stale managed artifacts are removed safely, and write-back is off by default.

The live seeder uses synchronous DataHub SDK/emitter writes to create a synthetic PII-tagged source, a two-hop lineage graph, owners, and a recorded retention query. This makes the judged flow reproducible without connecting to a private stack.

Challenges we ran into

The first green implementation still had unsafe edges. Its compatibility view became invalid after contract, non-null divergence escaped validation, dashboards could receive invalid field-lineage calls, pagination could silently truncate evidence, and textual query replacement could rewrite literals or unrelated columns.

Independent review converted those into regression tests. The final lifecycle rebuilds its view, validates with IS DISTINCT FROM, treats non-datasets as terminal field impacts, fails closed on inconsistent totals, uses an SQL AST, and distinguishes safe patches from review-required queries.

Accomplishments that we are proud of

  • A full expand/migrate/validate/contract lifecycle that executes in DuckDB.
  • Meaningful DataHub usage across schema, multi-hop column lineage, tags, owners, observed queries, and explicit document write-back.
  • Deterministic snapshot and live modes behind one compiler interface.
  • Fail-closed evidence handling and 26 local tests, plus a pinned official-package CI contract job.
  • Checked-in mergeable sample artifacts and a self-contained visual report.
  • Two independent review passes with every priority code finding resolved.
  • A public end-to-end DataHub Core v1.6.0 validation with live lineage, PII tag, owners, recorded query, browser evidence, and Decision document write-back/read-back.

What we learned

Metadata grounding is not just retrieval. A safe code generator needs completeness semantics, explicit trust boundaries, and a decision policy that treats missing context differently from no impact. DataHub's graph becomes most useful when the output also contributes durable context back for the next workflow.

What's next for SchemaFlight

  • Add dialect strategies and database-specific dual-write mechanisms.
  • Extend the request model beyond column renames.
  • Resolve table aliases with DataHub query subjects for more safe AST patches.
  • Add pull-request annotations and owner acknowledgement workflows.
  • Contribute reusable lineage-completeness and migration-context patterns upstream.

Built With

Share this project:

Updates