Twin

Twin is chaos engineering for data platforms. It reads your Datahub graph, breaks a real copy of your warehouse, checks whether the catalog predicted the damage and writes the verified truth back.

Comment

Inspiration

Every data catalog is a pile of confident claims. This table feeds that dashboard. This column is owned by that team. This asset is critical. Nobody ever checks. The claims are written once, drift quietly for months, and are believed anyway — right up until an upstream column goes null at 3am and it turns out the thing everyone assumed was safe was load-bearing. We now hand those same claims to AI agents and let them act on them.

Nobody builds the Chaos Monkey equivalent for a warehouse, because breaking production to find out what depends on what is obviously insane. But it isn't insane if you break a copy. DataHub made this buildable for the first time: the context graph already knows what is connected to what, down to the column. That's the map. What was missing was the experiment.

What it does

Twin is chaos engineering for data platforms. It reads your DataHub graph, breaks a real copy of your warehouse, checks whether the catalog predicted the damage, and writes the verified truth back.

Five stages, and every one of them runs for real:

  1. Read. Twin materialises an immutable estate graph exclusively through the official DataHub MCP server — datasets, table lineage, column lineage, ownership, queries. No side-channel reads into the metadata store.
  2. Predict. A deterministic propagation engine simulates the loss of each asset and ranks fragility. On the commerce estate raw_pg.fx_rates ranks first at 61.517 — an exchange-rate lookup nobody would nominate, sitting under revenue reporting.
  3. Break. A scenario executes against a disposable copy of the warehouse. Not mocked. Twin drops a real source column, changes a real type, drops a real relation, freezes real data, revokes real grants — and captures the error the warehouse actually returns.
  4. Grade. The observed blast radius is compared against the predicted one, and Twin scores its own catalog: precision, recall, and the specific assets it got wrong, by name.
  5. Write back. Verified results return to DataHub as seventeen structured properties per asset, plus context-confidence scores and incidents raised only for failures actually observed. One property records whether Twin has broken that asset for real or is ranking it on inference alone — so the catalog distinguishes tested context from assumed context. make prove-writeback reads all of it back over MCP to prove it landed.

Then it picks the next experiment. Executing one lowers its own verification novelty, so the queue reorders on evidence: after running fx_rate_column_drop, that scenario fell to third and owner_departure — targeting the top-ranked asset, still unverified — rose to first.

It also produces reviewable repairs: make repair emits a Markdown rationale plus a unified diff against the target dbt sources.yml. It never writes to your repo or calls GitHub — a human reviews, applies, re-ingests, reruns, and checks the graph fingerprint. Proposed, inspectable, reversible.

How I built it

The whole system runs on one machine under Docker Compose: DataHub GMS and frontend, MySQL, OpenSearch, Kafka, a Postgres warehouse, and Twin itself. No cloud account, no paid service, roughly 6 GB of RAM.

An estate worth testing. A three-table toy would have proved nothing, so the demo estate is a deliverable in its own right. Generated seed data flows through dbt into a real Postgres warehouse, is ingested into DataHub through the standard Postgres and dbt sources, then exercised with synthetic query workloads so usage-based signals are real. Commerce is 66 datasets, 125 table edges, 322 column edges.

Proof it wasn't overfitted. The obvious criticism of a single-estate demo is that the engine was tuned to that graph. So Twin runs against a second, independently designed estate: operations, a logistics platform — ERP and telemetry sources, facility and carrier models, its own ownership, its own workload of 1,197 recorded query executions, a separate DataHub platform-instance namespace. Same graph model, same propagation engine, same scorer, same verifier, same write-back code, zero estate-specific branches. ops_erp.shipments ranks first at 43.789.

Trustworthiness. Scoring output is byte-identical across runs — every dependency is pinned, because a floating library that changes traversal or serialisation order silently breaks determinism. make gate runs structural checks, determinism checks, and the 185-test suite, gating local pushes and GitHub Actions. reports/LATEST.md is rendered from captured artifacts and append-only history, never recomputed at render time, so the report cannot quietly disagree with the run it describes.

Challenges I ran into

The MCP server only reads. All six tools are reads. Twin needs to write verified results back, which meant building the write path on the DataHub SDK while keeping every read strictly on MCP — so the read side stays honest about what an agent can actually see.

Column lineage doesn't arrive where you'd expect it. Landing-field shape in the MCP lineage response cost real debugging time, and it's the difference between column-level and table-level blast radius.

Usage statistics aren't exposed over MCP at all, so the workload signal had to be reconstructed from query history.

Structured-property deletion collides in a way that makes cleanup non-obvious — which matters, because make unwrite has to fully undo everything Twin wrote to someone's catalog.

All five gaps were filed upstream with reproductions rather than assertions — two new issues (mcp-server-datahub#197, datahub#18974) and evidence attached to three existing ones. Every one came out of a run that failed, not a reading of the docs.

Then Twin caught me with my own thesis — twice in one day. The estate verifier's entity searches were unscoped: correct code, right up until a second estate landed in the same DataHub instance and commerce started counting 91 datasets instead of 66. Hours later, a second one: the write path computed context confidence without reading the campaign evidence ledger, so every asset in both estates published verification=0.00 no matter how many experiments had actually run. The ranking read that ledger correctly, which is exactly what hid it — selection behaved as designed while the catalog reported that nothing had ever been verified.

Neither was a broken estate. Both were claims about the estate that had quietly stopped being true, and neither was visible to the test suite. I only found the first because the nightly writes down its failures.

Which is the hardest problem here: honesty. Twin's adversarial source-column scenario measured 0.69 precision and 1.00 recall, with five named false alarms. An afternoon's tuning would have produced a clean 1.00 to screenshot. I kept it and shipped it in the README. The over-prediction is caused by thin catalog context — that is the finding. A tool built to measure whether metadata survives contact with reality does not get to hide its own error bars. Same principle produced examples/history/attempts.jsonl, the file that caught the bug above: a success-only history can't distinguish a failed night from a night nobody ran.

Accomplishments that I'm proud of

  • A genuinely closed loop — DataHub context in, real warehouse execution in the middle, verified context back out, proven by MCP read-back on 66 commerce and 25 operations assets carrying all seventeen properties.
  • Cross-estate generalisation on two independently designed graphs with no estate-specific code.
  • Verification against a live warehouse, not a simulation of one.
  • A reported precision below 1.00, on purpose.
  • A catalog that distinguishes tested context from assumed context — and an experiment queue that reorders itself on that evidence.
  • Five upstream findings contributed back to DataHub and its MCP server.
  • A failure history that actually caught something — a scoping bug in Twin's own verifier, found the day before submission.
  • 185 passing tests and a five-check quality gate that runs without DataHub or the warehouse.

What I learned

The dangerous metadata bug is almost never a wrong value. It's a query that quietly changes meaning when the world around it changes — the unscoped search that was right until a second estate existed, the derivation that silently omits its evidence. Nothing alerts on either. Only executing against reality catches them, which is the whole argument for building Twin and, it turns out, for pointing Twin's own discipline back at Twin.

Agents inherit that uncertainty silently. An agent reading DataHub gets no signal about which parts of the graph have ever been tested. That is exactly why Twin publishes context confidence as first-class properties: so the next agent to read the graph can tell verified context from assumed context — and so an asset ranked first on inference alone says so, in the catalog, where an agent would look.

What's next for Twin

Continuous verification against production catalogs — the nightly already runs, ranks, executes, and appends history unattended. Widening the scenario library beyond the current fault classes. Publishing generated repairs as normal reviewed pull requests. And a read-back assertion in the nightly: three bugs this week were published-claim bugs that the quality gate could not see, because nothing checked that a published property still agreed with the evidence it came from.

Built With

Share this project:

Updates