Premortem — how it breaks, before you merge

Before you merge a column rename, Premortem scores which DataHub consumers break and how, and writes the forecast back to the catalog — before a dashboard tells the business

Comment

Inspiration

I am a data engineer about to rename a column. DataHub's Impact Analysis already lists what is connected. What it cannot tell me is which consumer queries actually break, or how badly — filter vs display-only vs false alarm. So I guess, ship, or stall. When the guess is wrong, it is rarely me who finds out first: a dashboard or a metric the business trusts breaks, and that Slack thread is the incident — for a change that was sitting in a PR while the answer was knowable.

Schema changes are human-initiated. I wanted the rehearsal before merge: replay what people actually run, score blast radius, write it back so the next person inherits it.

For judges — fastest paths in

Path Link Time
Watch the demo Demo video ↗ ~3 min
Project page Project page ↗ 60 s
Try it locally (CLI) README → Try it locally — premortem + premortem gate on the bundled corpus ~2 min
Measure the frozen eval pip install -e ".[dev]" → python eval/run_eval.py → eval/RESULTS.md ~1 min
Wire the MCP (Claude / Cursor) MCP setup guide ↗ ~5 min + local DataHub
Live rehearsal report HTML report ↗ 30 s

TL;DR: Bind query history → HARD / SOFT / UNKNOWN / CLEARED → repair → CI gate → Quality write-back. Three doors, one binder: CLI · MCP · eval. Frozen numbers 0.97 / 1.00 / 0.00. Dual-MCP compose with DataHub — not a UI fork.

Why it matters

Without Premortem, Impact Analysis answers “what is connected?” and I still guess on merge. With it, I get a merge decision (ship / patch first / hold for UNKNOWN), a CI gate so HARD and unread SQL cannot silent-pass, and a catalog memory on the Quality tab so the next engineer inherits the forecast instead of learning from a broken dashboard. Measured, not hand-waved: decoy false alarms 0.50 → 0.00 with binding; HARD precision 1.00 vs 0.40 if every dependent “breaks”; 22/22 eligible repair patches on the frozen corpus. Usability for a platform team: offline eval in under a minute, drop-in premortem gate, and one agent session that composes with DataHub — not a UI fork to adopt.

Who Premortem is for and the five-step journey from PR to catalog write-back

Primary user: the engineer owning the table. Secondary: the platform engineer who later opens the dataset and inherits the forecast. Integration model: compose with DataHub via MCP — not a UI fork.

What it does

Premortem takes a proposed column rename or drop, reads SQL from DataHub query history, binds each column reference to a real table, and issues a per-query verdict under the Impact Analysis baseline.

Interactive product walkthrough ↗ — seven beats on the project page: Impact Analysis → rehearse → bind → refuse → repair → gate → notify + write-back.

Verdict Meaning
HARD Column in WHERE / JOIN / GROUP BY / ORDER BY / HAVING / window — careless repair can change results
SOFT SELECT-list only — mechanical rename
UNKNOWN Unparseable or still ambiguous after binding — needs a human; I do not guess
CLEARED Named subset of UNAFFECTED: same column name, but it binds to a known different table — false alarm, listed

The differentiator is binding, not name matching. Two tables can both have order_status; a name grep flags the wrong one. Binding clears the false alarm when the qualifier resolves to a known non-subject table — and refuses (UNKNOWN, no patch) when an alias is shadowed or a qualifier cannot be resolved confidently.

Two tables carry order_status; name matching flags the wrong one; binding clears it

Binding took decoy false positives from 0.50 to 0.00 on the frozen eval. (Verdict taxonomy and dual-MCP flow are in the gallery — not repeated here.)

Rehearse → repair → gate → notify

The forecast is not the end of the workflow:

  1. Rehearse — binder scores each query (HARD / SOFT / UNKNOWN / CLEARED) against DataHub context.
  2. Repair — for subject-bound HARD/SOFT hits, emit a SQL patch (order_status → order_state). CLEARED, UNKNOWN, SELECT *, and ambiguous binding are refused — I do not invent fixes. Round-trip on the frozen corpus: 22/22 eligible patches. Samples: examples/patches/.
  3. Gate — premortem gate fails CI when findings meet --fail-on. Drop-in workflow: examples/ci/premortem-gate.yml.
  4. Notify — surface who to warn (dataset owners from the catalog) alongside the forecast.
  5. Share — same run can emit --html-out for reviewers who will not open the Quality tab (markdown + JSON still available).

The gate is the part with teeth. A forecast is advisory — someone still has to read it. premortem gate runs the same binder inside CI and exits non-zero when the change would ship a HARD break, so the rehearsal becomes policy instead of a document: no GitHub App, no service to run, no catalog required — a workflow file, a directory of SQL, and an exit code. The default is deliberately strict. --fail-on hard,unknown means SQL the binder could not read blocks the merge rather than passing quietly, and asking for --fail-on hard alone exits 2 — “refusing silent green” — instead of letting unparseable input through. That is the same refuse-instead-of-guess rule as the classifier, enforced at the one point where it costs something.

In one Claude Code session I register DataHub's MCP server and Premortem's. Plain language: rehearse renaming order_status on ORDER_HISTORY. Live reads use the DataHub Agent Context Kit (schema, lineage, query history). Premortem returns the forecast plus a write_payload. The host applies that payload as-is: tag + description via DataHub's MCP mutation tools; the custom assertion via GraphQL — OSS MCP Data Quality tools are DISABLED (the gap I filed as #151). The forecast lands as platform="premortem" on the Quality tab so the next person inherits it.

The numbers

I committed forty hand-labeled queries and gold labels before the classifier could pass them (5709e88). The binder was developed against tests/ fixtures; the frozen forty were never tuning targets. Three independent blind LLM labeling runs (given only the corpus, schema fixture, and taxonomy — not the rest of the repo) agreed with the authored labels 40/40.

run accuracy HARD prec HARD rec decoy FP
Premortem (binder) 0.97 1.00 0.94 0.00
every-dependent-breaks 0.40 0.40 1.00 1.00
substring grep 0.45 0.44 1.00 0.83

Accuracy 0.97 = 39/40 (truncated). HARD precision n=15; decoy FP n=6. Chart vs baselines is in the gallery.

Two unflattering results, published on purpose. A heuristic adjudicator bound 3 of 9 UNKNOWNs with bind accuracy 0.00 — shipped disabled. An LLM on the same residue correctly declined all three (gold is UNKNOWN); no lift, so live stays binder-only. Named miss: q39 (hard→soft). Cache committed so B2 reproduces with no API key.

pip install -e ".[dev]"
pytest -q
python eval/run_eval.py    # 0.97 / 1.00 / 0.00 — offline; no live catalog required

Foreign SQL (no labels): 118 queries from mozilla/bigquery-etl — parse rate 84.75%, 86% UNKNOWN under missing schemas (narrowable refs would bind with schemas present; the 11% CLEARED are qualified refs to known non-subject tables). Report ↗

Reproduce it

Judges do not need a hosted demo. Clone the public repo and run the product locally — Python 3.11+, no DataHub required for the CLI path:

git clone https://github.com/prasadt1/premortem-datahub.git
cd premortem-datahub
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Same binder as the video — CLI rehearsal
premortem --queries-dir eval/corpus --rename order_status:order_state \
  --subject-table order_history --tables-json eval/schema.json

# Merge gate — expect exit 1 (HARD / UNKNOWN would ship)
premortem gate --queries-dir eval/corpus --rename order_status:order_state \
  --subject-table order_history --tables-json eval/schema.json --fail-on hard,unknown
echo "exit=$?"

Full README section: Try it locally (CLI).

Optional — measure the frozen eval (published numbers, not required to try the tool):

  1. pytest -q
  2. python eval/run_eval.py — expect 0.97 / 1.00 / 0.00 → eval/RESULTS.md

MCP door (Claude Code, Cursor, or any MCP host): needs a local DataHub with query history. Config + prompt: examples/MCP_COMPOSITION.md.

pip install -e ".[datahub,mcp]"
# optional demo seed: python tools/seed_demo_environment.py

Register Premortem + DataHub MCP, then ask: Rehearse renaming order_status to order_state on ORDER_HISTORY.

Live CLI against your catalog: premortem --live --rename order_status:order_state --html-out rehearsal.html (subject needs query history — #18676).

Premortem is a library + MCP server that composes with DataHub — not a hosted portal. Three doors, one binder: CLI · MCP · eval. Video shows MCP; the commands above are CLI; run_eval.py is the measured freeze.

How I built it

Layered architecture: three entry points, catalog-agnostic binder core that composes with DataHub via CatalogClient

One binder, three doors in, DataHub via CatalogClient — the catalog-agnostic core is why the frozen eval runs offline.

Beyond what the diagram shows:

  • Binding, not name matching — resolve unqualified columns against every in-scope table schema; ≥2 candidates → UNKNOWN; CLEARED only when the qualifier resolves to a known non-subject table.
  • Refuse instead of guess — alias shadowing, unresolvable derived-table qualifiers, and unparseable input become UNKNOWN / gate-fail; never a confident-wrong CLEARED or a corrupting patch.
  • Repair on a shared seam — the same binder decisions drive patches; round-trip kill criterion 100% on eligible corpus queries.
  • Merge gate — offline or live CI exit codes (0 clean / 1 blocked / 2 misconfigured); default fail-on includes UNKNOWN so unread SQL cannot green-light, and the tool refuses a --fail-on that would let unparseable SQL through.
  • Write-back as a DataHub object — tag + description via DataHub MCP; assertion via GraphQL (OSS MCP DQ disabled — #151); Quality tab category with zero UI code.

Challenges I ran into

  • Tautological adjudicator looked smart, scored 0.00 — frozen eval caught it.
  • Live path once ran without sibling schemas while eval had them — green tests, wrong demo.
  • Lineage self-edge nearly became a fake “1 downstream” baseline on camera.
  • Adversarial review after the freeze found alias-scope binder bugs that would have shipped a confidently-wrong “safe to merge” — Tier-1 fix: refuse those shapes; disclosed as corpus blind spots in eval/RESULTS.md.
  • OSS gaps filed upstream: #18674 column-level assertion run events, #151 DQ tools off on MCP, #18675 Document search, #18676 listQueries indexing lag.

Accomplishments that I'm proud of

  • Benchmark pinned before it could be passed — the frozen forty were never tuning targets
  • Turns Impact Analysis from a dependency list into a merge decision + catalog memory the next person inherits
  • The forecast is enforceable, not advisory — premortem gate blocks the PR on the same binder, and refuses to be configured into a silent pass
  • Zero decoy false alarms on the frozen eval (decoy FP 0.00)
  • Two negative agent results published instead of deleted (heuristic adjudicator 0.00 bind accuracy; LLM correctly declined all three UNKNOWNs with no lift)
  • Repair round-trip 22/22 on eligible patches; unconfident shapes refused rather than guessed
  • Four upstream DataHub / MCP issues filed from real gaps hit while building

What I learned

  • Classification without binding is just name matching
  • An agent that always says yes is not an agent
  • Verify the demo path separately from the test path — green tests once hid a wrong live demo
  • When the binder is unconfident, refuse — publishing the blind spot is worth more than hiding it behind 0.97

What's next for Premortem

  • Per-scope binding via sqlglot's scope resolver (today those shapes are refused, not guessed)
  • Event-driven rehearsal through DataHub Actions
  • ML-consumer entities and type-change blast radii
  • LLM-on-residue stays measured (B2) and non-default until it lifts the frozen eval
  • Shareable HTML report — shipped as --html-out; PR-check summary from gate JSON is a follow-on

Links

Evidence Link
Demo video Demo video ↗
Project page Project page ↗
Product walkthrough product-tour ↗
Live rehearsal report Report ↗
Source github.com/prasadt1/premortem-datahub
Eval results eval/RESULTS.md
Repair samples examples/patches/
Merge gate (CI) premortem-gate.yml
Dual-MCP setup MCP setup guide ↗
Eval explorer eval-explorer
Real-world run real-world-run
Freeze commit 5709e88
Upstream issues OSS_ISSUES.md

Honest scope. The video demo uses a constructed Quickstart (synthetic shipments, seeded queries/lineage). Accuracy numbers come only from the frozen eval — which contains no cross-scope alias shadowing, derived-table aliases, or DML; those classes are covered in tests/ and refused rather than guessed. Macros and dynamic SQL land in UNKNOWN; no query evidence ≠ safe to change. Premortem reasons about tables, not view expansion.

Built With

Share this project:

Updates