Rehearsal

A DataHub-native change agent that turns live metadata into approval-bound, rollback-proven PostgreSQL column renames.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Rehearsal

Inspiration

A column rename looks small in DDL and large in production. Impact tools can list downstream assets, and code generators can emit plausible SQL, but neither proves that the plan came from current catalog evidence, the accountable owner approved that exact plan, old and new consumers remain compatible, and rollback preserves the governed source.

Rehearsal closes that gap. It turns DataHub metadata into a reviewable compatibility change whose rollback has actually run.

What it does

Rehearsal accepts a proposed PostgreSQL column rename and reads schema, ownership, governance, assertions, column-level lineage, and known queries through DataHub Agent Context Kit.

It blocks incomplete or inconsistent context, calculates deterministic risk, and compiles an additive compatibility view plus exactly seven review artifacts. The request, catalog context, policy decision, and artifact hashes are frozen into one packet digest before owner approval.

After approval, Rehearsal refreshes the DataHub evidence and blocks execution if anything drifted. It recompiles the packet, reconstructs the full catalog schema in an isolated semantic environment, replays supported DataHub queries, inserts a new source row to prove the compatibility view sees ongoing writes, and executes rollback by removing only that view. It then confirms that the source relation, source column, and source rows remain intact.

In live mode, successful verification writes the exact Decision proof and a verified tag back to DataHub, then confirms both through fresh read sessions. Rehearsal never emits or executes destructive contract SQL.

Why it is different

This is not another metadata chat interface or blast-radius dashboard. Rehearsal converts context into execution boundaries:

  • Catalog evidence is frozen under a digest.
  • Approval is bound to the owner, request, context, policy, and all seven artifact hashes.
  • Execution recollects DataHub context and stops on drift.
  • Known queries are replayed against the source and compatibility view.
  • A post-plan source write must remain visible.
  • Rollback must execute and preserve the governed source.
  • Proof is written back and reread through fresh DataHub sessions.

Every green check in the demo comes from the running backend, not a scripted animation.

How we built it

The core is Python 3.11/3.12 with Pydantic contracts, DataHub Agent Context Kit and SDK, SQLGlot validation, SQLite semantic rehearsal, FastAPI, and Typer. The judge interface is React, TypeScript, and Vite. DataHub OSS 1.6.0 supplies the live metadata graph.

GitHub Actions validates formatting, linting, strict typing, tests, packet compilation, the production UI, Python distributions, and installed-wheel smoke on both supported Python versions. GitHub Pages is clearly labeled as a captured-evidence review surface, while the narrated video records a real live-mode run.

Challenges we ran into

The difficult part was preserving one identity across catalog reads, generated artifacts, human approval, execution, rollback, and write-back. Rehearsal solves that with deterministic digests and a mandatory post-approval context refresh.

DataHub OSS 1.6 also exposed real integration edges: PostgreSQL interval expressions needed normalization inside the SQLite rehearsal, and fresh Document verification required a same-client SDK fallback when Agent Context returned only the entity URN. Both paths are covered by regression tests.

Accomplishments that we're proud of

The recorded run proves the complete loop:

  • 3 downstream datasets discovered from column lineage
  • 3 catalog queries replayed
  • 7 additive migration artifacts frozen under one packet digest
  • a new source write visible through the compatibility view
  • rollback executed with both source rows retained
  • destructive contract execution remaining false
  • Decision and verified-tag write-back confirmed by fresh DataHub reads

The public repository is Apache-2.0 licensed and ships full setup instructions, sample artifacts, a deterministic fixture path, and a live DataHub path.

What we learned

Metadata becomes operationally valuable when it controls whether code may execute, not merely what an agent says. The strongest agent is not the one that generates the most SQL. It is the one that can prove why a change is allowed, what ran, what did not run, and whether rollback preserved reality.

What's next for Rehearsal

The next product step is to deliver the packet as a pull request with warehouse-specific adapters while retaining the same evidence, approval, and rollback contract. dbt models and additional SQL engines can plug into the compiler without weakening the proof boundary.

Built With

Share this project:

Updates