RippleProof

• 

  Five ways a confident agent gets this wrong. Across a checked corpus of 14 cases, 8 of which must refuse, it made 0 false repairs.

• 

  Executed, not linted: four dbt projects build in isolation proving the exact models ran, and the rename rolls back in a throwaway database.

• 

  The run, including the three times it stopped: an ambiguous model mapping, unproven dashboard column use, and lineage that moved.

• 

  One rename, five downstream assets: four dbt models repaired from column lineage, one dashboard held back because its use is unproven.

• 

  Every captured run, committed and readable end to end. They differ on purpose, and one of them fails its dbt build.

Inspiration

A schema migration can be syntactically correct and still break a dbt model, a dashboard, a contract, or a team in another repository. Code generators see the file in front of them. DataHub sees the organizational graph around it.

RippleProof makes that metadata a required input to code generation rather than an after-the-fact debugging aid. Impact analysis is where it starts, not what it outputs.

What it does

Given a DataHub dataset and a PostgreSQL column rename, RippleProof reads the source schema, column-level lineage, wider asset reachability and ownership over the DataHub MCP server, then generates the dbt SQL and contract repairs that the evidence supports across every affected repository, builds each one in an isolated sandbox, applies and rolls back the migration in a disposable networkless PostgreSQL container, and opens review-only pull requests with auto-merge disabled.

It never executes the migration against a source database, never edits your repositories while planning, and never merges anything.

Five ways a confident agent gets this wrong

The interesting part is where it stops. Each of these is a case where automation finishes cleanly, reports success, and leaves something silently broken.

Those five, plus nine more, run as a deterministic corpus on every change: 14 of 14 cases behave as declared, 8 of them exist only to check it still refuses, and 0 false repairs. That is a regression suite over cases we authored, not an industry benchmark.

The campaign you can inspect

One campaign renames analytics.orders.user_id to customer_id, bound to manifest SHA-256 7b907e81...135134a31:

• 5 downstream assets found, 4 with column-level evidence
• 8 files generated across 4 repositories, 4 SQL model repairs and 4 dbt contract updates
• 4 dbt projects built in isolated DuckDB roots, each proving the exact expected models ran
• 1 Looker dashboard held for manual review, because reachability is not usage
• 4 review-only pull requests opened with required checks passing and auto-merge disabled
• 6 assets carrying an approved-for-review decision, read back through a fresh DataHub client, explicitly not approved-to-merge

Everything above is readable at the hosted site with no account, install, or service. One of the captured runs fails its dbt build on purpose and is kept in the gallery rather than removed.

How we built it

Python 3.11, the DataHub MCP server over stdio for reads, the DataHub Python SDK for the decision write-back, dbt Core with dbt-duckdb for isolated validation, a disposable PostgreSQL container for the rename and rollback, and the GitHub API for review-only delivery.

The SQL transformer is deliberately narrow: a conservative lexical transformer behind a structural ownership preflight. Before rewriting a single identifier it refuses any model whose column ownership it cannot prove, including joins, CTEs, set operations, subqueries, lateral relations, and anything that is not a plain SELECT. Strings, comments, dollar-quoted bodies and Jinja are preserved rather than rewritten. It repairs the narrow case it can prove and hands back the rest.

Challenges

Cached lineage outlived the graph. The MCP server calls searchAcrossLineage without skipCache, and its tool schema exposes no cache-bypass option. We hit this in a controlled test: an edge was removed, the cached interface still returned it, and an independent cache-bypassed read did not. RippleProof refuses rather than execute yesterday's plan against today's warehouse. We upstreamed a narrowly scoped fix as mcp-server-datahub#190.

Read-your-own-writes. DataHub acknowledges a write before the search index reflects it, so ingest-then-search in a single run intermittently misses fresh data. Our readback goes URN-direct rather than through search.

Proving a build actually built. dbt build exits 0 when its selector matched nothing. A green build that proved nothing is not evidence, so each validation asserts the exact expected node set rather than the exit code.

What we learned

Verification is a different discipline from generation. Every automated check can pass on output that is confidently wrong, so the useful gates turned out to be the ones that assert what should have happened rather than that nothing errored.

What's next

Wider change types beyond renames, more source platforms, and a proper answer to partial lineage coverage. Absence from DataHub is not proof that no other dependency exists, and the tool says so rather than implying otherwise.

Built With

• acryl-datahub
• ai-agents
• column-level-lineage
• data-lineage
• datahub
• datahub-mcp-server
• dbt
• dbt-core
• docker
• duckdb
• github-actions
• github-api
• mcp
• openai
• playwright
• postgresql
• python
• schema-migration
• sqlglot-free
• structured-properties