Ripplex

RippleX doesn't just flag risky schema changes - it fixes them: reads DataHub lineage, rewrites SQL, opens real PRs, proves correctness live, then writes assertions and playbooks back into DataHub.

Comment

RippleX — Project Story

Inspiration

Picture the moment: a column needs to change — order_total on the orders table, FLOAT to DECIMAL(12,2), a one-line migration any engineer would call routine. Someone runs a lineage query, finds four dbt models, migrates them, ships it. Three weeks later the monthly revenue dashboard is off by tens of thousands of dollars and nobody can explain why — until someone finally traces it to a BI chart that read straight from the raw table, never touching a single dbt model, invisible to any lineage query that only knows how to look one layer deep. Then it happens again, this time surfacing in a churn-prediction model whose training feature quietly drifted the moment the column's precision changed.

This isn't a rare failure mode. It's the default failure mode. Lineage tools show you the dbt models. They don't show you the dashboard that reads the whole table without ever mentioning the column by name, or the ML feature table quietly built on top of it. So the change ships, something breaks three hops downstream weeks later, and the post-mortem ends the same way it always does: "we didn't know anyone was using that."

For a long time the instinct is to fix this with better documentation — a wiki page, a migration checklist, a pinned Slack thread. None of it survives contact with a deadline. The problem was never that the knowledge didn't exist. The problem is that knowledge doesn't rewrite the nine files that depend on it, and it doesn't prove anything to anyone after the fact.

That's the first thing RippleX became: an agent that doesn't just know the blast radius of a change — it finds every real consumer, deterministically, and acts on what it finds.

Then came the second realization. Finding the blast radius solves half the problem. The other half is trust: once an agent starts rewriting other people's SQL and touching a live warehouse, how does anyone — a teammate, a judge, a future on-call engineer — verify it did the right thing, and keep believing that days or months later? A green checkmark on a pull request isn't enough. The proof needed to outlive the migration itself: a real, permanent, independently-checkable object sitting inside the same catalog everyone already trusts.

That's why RippleX doesn't verify a migration once and forget — it writes a standing DataHub Assertion for every consumer it touches, pass or fail, so the proof is still sitting in DataHub's own Validation tab the day someone goes looking for it.

Finally, there was the memory problem. The same shape of migration — a FLOAT retyped to a DECIMAL, a column deprecated in favor of its replacement — happens over and over across an organization, and every time the agent (or the human) starts from zero, re-deriving reasoning it already proved correct the last time. DataHub's Agent Context Kit and MCP Server made something click: the catalog itself, with its document graph and saved institutional knowledge, could be the agent's long-term memory, not a context window that evaporates the moment the request ends.

The result is an agent that doesn't just analyze a migration — it maps, migrates, verifies, remembers, and proves. Not a tool that tells you what might break. Something that answers the question no lineage dashboard can:

"If I approve this migration — what actually depends on it, will it still be correct afterward, and can I prove that to someone who wasn't in the room?"

What It Does

RippleX is an autonomous data-migration agent that takes a schema-change intent — typed into its dashboard, or heard live from a real DataHub MetadataChangeLog event — and runs a full six-agent pipeline against a real warehouse and a real DataHub catalog before anything is allowed to ship.

The Six Specialist Phases

Phase Agent What It Does
1 Cartographer Traverses DataHub's real column-level lineage in two stages — column-level first, then a table-level fan-out that catches BI dashboards and ML feature tables lineage alone would miss — and scans every consumer for missing owners/descriptions and at-risk production ML models
2 Planner Sequences the work into an expand → migrate → verify → contract plan, so nothing migrates before its own upstream
3 Surgeon Rewrites every consumer's real SQL with SQLGlot — never a string replace — and generates a companion schema.yml doc/test update for the same pull request
4 Verifier Proves the whole multi-hop chain correct with one self-contained chained query per consumer, run against the real warehouse
5 Scribe Drafts the migration playbook — but first checks DataHub's own document graph for a prior playbook covering the same type-transition pattern, and skips re-deriving prose from scratch on a hit
6 Reviewer Assembles everything into a package and gates every real mutation — PRs, warehouse writes, catalog writes — behind an explicit human approval call, enforced in code, not convention

What a Real Run Looks Like

This is a real transcript, from a live run against the deployed service — every number below is a real, reproducible result, not a mock:

POST /intake   { target: raw.orders.order_total, FLOAT → DECIMAL(12,2) }

CARTOGRAPHER   21 real downstream consumers found (2-stage lineage traversal)
               18 owners missing · 18 descriptions missing   [Metadata Debt Radar]
               1 production ML model reached: churn_prediction_model  [ML Blast-Radius Guard]
SURGEON        6 real SQL diffs staged, each with a companion schema.yml
SCRIBE         memory hit — recalling a prior migration's verified approach

⚠ AWAITING HUMAN APPROVAL — nothing above has mutated anything yet

POST /migrations/{id}/approve

EXECUTOR       6 real, ordered pull requests opened against ripplex-sample-estate
VERIFIER       6/6 chained parallel-run verifications passed — 0 mismatches
               6/6 real DataHub Assertions provisioned — status: COMPLETE
               18 owners + 18 descriptions fixed, for real
               1 real warning written onto churn_prediction_model's DataHub page
SCRIBE         playbook written back, linked to the recalled prior playbook
REVIEWER       contract closed — raw.orders.order_total deprecated, 90-day sunset

✅ RUN COMPLETE — every line above points at a real, independently checkable object

RippleX Doesn't Just Verify — It Writes Back

Most migration tooling produces a report and stops. RippleX takes autonomous action through real DataHub and Git write-backs:

Action What It Does
provision_assertion Turns a one-time parallel-run proof into a permanent DataHub Assertion — pass or fail — visible in DataHub's own Validation tab long after the migration ships
deprecate Writes a real deprecation aspect (sunset date + replacement pointer) onto the old column
save_playbook Writes the migration playbook back into DataHub as a real document, linked to RippleX's own registered Application identity and to any prior playbook it recalled
own / describe Fixes real ownership and documentation gaps found while walking the blast radius — the catalog comes out healthier than it went in
describe (ML-targeted) Writes a real, standing warning directly onto an affected production model's DataHub description
open_pr_for_diff Opens one real, ordered, mergeable GitHub pull request per consumer, carrying the SQL diff and a companion schema.yml
DataHub Action (systemd) Subscribes to real MetadataChangeLog events and calls RippleX automatically the moment a real schema change happens

Three Ways to Trigger

1. Live dashboard (zero setup) — the hosted dashboard needs no API key, no ANTHROPIC_API_KEY, nothing to configure. Click through, watch it run.

2. A real DataHub event (automatic) — a DataHub Action running as a systemd service on the same VM as DataHub itself subscribes to real MetadataChangeLog events and calls RippleX's webhook the instant a real schema change happens:

DataHub MetadataChangeLog → RippleX Action → POST /webhook/datahub

3. Direct API (programmatic):

curl -X POST https://ripplex-854441956422.us-central1.run.app/intake \
  -H "Content-Type: application/json" -H "X-API-Key: $RIPPLEX_API_KEY" \
  -d '{"target_urn": "urn:li:dataset:(...)", "target_column": "order_total",
       "old_type": "FLOAT", "new_type": "DECIMAL(12,2)"}'

How We Built It

The Agent Core — LangGraph, with a real deterministic fallback

RippleX is a LangGraph-orchestrated pipeline built around one hard architectural rule: the LLM only ever proposes; deterministic code is the only thing that ever mutates anything. Lineage traversal, SQL rewriting, verification execution, and every DataHub write-back are 100% deterministic Python. Only judgment calls — is this breaking, what's a good playbook sentence — go through an LLM.

A key architectural decision: even those judgment calls needed a real, non-mocked fallback. Judges testing the live demo shouldn't need to hunt down an ANTHROPIC_API_KEY for RippleX to do genuine work, so ripplex/llm/factory.py swaps in a deterministic provider when no key is configured — SQLGlot-derived verification parameters, a conservative breaking-change default, a templated playbook built from the same real structured data an LLM prompt would have used. Every fallback response says in plain text that it's a fallback. Everything around it — lineage, SQL rewriting, warehouse verification, PRs, catalog writes — is fully real either way.

human intent ──▶ FastAPI (Cloud Run) ──▶ LangGraph pipeline (Cartographer →
 or DataHub        /intake  /migrations       Planner → Surgeon → Verifier →
 webhook event      /approve  /metrics)        Scribe → Reviewer)
                          │
                          ▼
              ┌── DataHub access layer ──┐        ┌──────────────────────────┐
              │  20 real MCP tools        │◀──────▶│  DataHub (GCE VM)         │
              │  column-level lineage     │        │  GMS + frontend + Kafka/  │
              │  emitter (Assertions,     │        │  OpenSearch/MySQL, plus   │
              │  deprecation, playbook)    │        │  a RippleX Action running │
              └──────────────────────────┘        │  as a systemd service     │
                          │                         └──────────────────────────┘
       ┌── SQLGlot rewriter ── warehouse executor ──┐
       │   chained parallel-run verification         │
       │   Git PR automation ── state + audit DB     │
       └──────────────────────────────────────────────┘

DataHub Integration — MCP Server, Agent Context Kit, and a real contribution

We went deep on DataHub itself rather than around it:

  • MCP Server — a typed façade over 20 real DataHub tools (ripplex/datahub/mcp_client.py), every one a verified pass-through, no business logic hidden inside it.
  • Agent Context Kit — wraps the real DataHubContext SDK for lineage, search, and document read/write.
  • DataHub Skills — we didn't just consume the ecosystem, we contributed to it: a real, open pull request (#59) packaging the expand → migrate → verify → contract recipe we discovered was actually necessary.
  • RippleX registers itself as a real DataHub Application entity, and every playbook it writes links back to that identity, so the catalog shows who wrote it, not just that one exists.

The Write-Back Layer — real aspects, not mocked calls

Assertions, deprecation, and ownership all go through datahub.emitter.rest_emitter.DataHubRestEmitter and MetadataChangeProposalWrapper, emitting real OSS aspect classes (AssertionInfoClass, AssertionRunEventClass, DeprecationClass) — not DataHub's higher-level Cloud-gated convenience wrappers, which require acryl-datahub-cloud and aren't available on the OSS deployment this project runs against. We found that split by introspecting the installed SDK directly rather than trusting documentation, and it shaped the entire write-back architecture.

The Architecture

Ripplex System Architecture

Infrastructure

Component Technology
Orchestration LangGraph — six-agent pipeline
Reasoning Anthropic Claude, with a real deterministic fallback when no key is configured
Backend FastAPI + Uvicorn/Gunicorn (Python 3.11)
Catalog integration DataHub — acryl-datahub, datahub-agent-context (Agent Context Kit), mcp-server-datahub (MCP Server), a real DataHub Action
SQL rewriting SQLGlot — real parsing, never a string replace
Warehouse & state PostgreSQL, SQLAlchemy 2.x + Alembic
Git automation PyGithub + GitPython
Hosting Google Cloud Run (service) + a GCE VM (DataHub + warehouse + Action) + Cloud SQL (state/audit)
CI/CD GitHub Actions — lint, typecheck, test → Docker build → Artifact Registry → Cloud Run

Cloud Run runs the stateless FastAPI service; a single GCE VM runs DataHub itself (GMS, Kafka, OpenSearch, MySQL, frontend) and the demo warehouse and a real DataHub Action running as a systemd service — three things sharing one already-paid-for box, a deliberate cost tradeoff against DataHub-on-GKE for a project that needs to be started and stopped repeatedly between work sessions.

Once the core pipeline worked, we asked a harder question: what would make a skeptical judge believe this is real, not staged? That produced the six features documented above — Living Contracts, Metadata Debt Radar, Compounding memory, the live activity feed, ML Blast-Radius Guard, and schema.yml generation — each one built specifically to leave a permanent, checkable trace, and each one landing RippleX in a different hackathon category: Agents That Do Real Work (the whole pipeline), Metadata-Aware Code Generation (Surgeon reads real schemas before generating real, mergeable migration code), and Production ML Agents (ML Blast-Radius Guard's real lineage hop from feature to model).

Challenges We Ran Into

1. The SDK lies about what the server will accept — three separate times

CustomAssertionInfo.field is typed as a plain string, but the server 422s unless it's a real schemaField URN. add_owners(ownership_type=None) is typed Optional, but the server 400s on an explicit None. Most dramatically: IncidentInfo.entities is typed List[str] with zero documented constraint, but the server flatly rejects an MLModel urn as an invalid relationship destination — "not a valid destination for field path: /entities/*" — a restriction that exists nowhere except in DataHub's own server-side schema. None of these were caught by unit tests with a mocked emitter; every one was caught only by driving real traffic against the redeployed, live service. We rewrote the ML Blast-Radius Guard's write-back mechanism entirely, mid-build, once we found the third one — switching from a standing Incident to a real warning written via update_description, which has no such restriction.

2. A real region ran out of room

Deep into final verification, GCE started rejecting every VM start with ZONE_RESOURCE_POOL_EXHAUSTED — and it wasn't a one-zone blip. We tested e2-standard-2 and e2-medium across all four us-central1 zones (a, b, c, f); every single one was exhausted, and Google's public status page listed no incident at all. We migrated the entire VM live: snapshotted the disk, recreated the instance in us-east1 (which had real capacity), re-reserved a static external IP there (regional resources can't cross regions), and redeployed Cloud Run pointing at the new address — without losing a single byte of DataHub's existing state.

3. Judge-friendliness vs. security

We wanted zero setup friction for anyone testing the live demo — no API key to hunt for. The honest fix was the real deterministic LLM fallback described above, not a fake "demo mode": every reasoning step still produces a genuine, structured, non-mocked result, clearly labeled as a fallback, while everything else stays fully real regardless of whether a key is configured.

4. Docker image built silently missing its own test fixtures

The deployed image produced diff_count: 0 on every real run for an embarrassingly simple reason: sample_estate/dbt_project — the real dbt project Surgeon needs to find consumers in — was never copied into the Docker image. Caught only by driving a real /intake call against the deployed container and noticing the consumer count was real (21) but the diff count wasn't. Fixed by having the deploy script clone the sample estate before docker build runs, and by adding a .dockerignore that had never existed before.

5. GitHub's rewritten history and stale infrastructure notes

Purging an unrelated internal decision log from the repo's git history (at our own request, mid-build) meant every downstream commit hash changed — a real, deliberate history rewrite via git filter-repo and a force-push, done carefully rather than casually, since it's the kind of operation that breaks any existing clone. Separately, moving the DataHub VM to a new region meant every piece of documentation that mentioned the old zone and IP had to be tracked down and fixed, not just the infrastructure itself — stale docs are their own kind of bug.

Accomplishments We're Proud Of

Nothing in this project is simulated. Every button on the live dashboard triggers a real pull request, a real query against a real warehouse, or a real write into a real DataHub instance — including the failure paths.

RippleX legitimately spans three of the four hackathon categories — Agents That Do Real Work, Metadata-Aware Code Generation, and Production ML Agents — without stretching the definition of any of them.

A real, open-source contribution back to datahub-project/datahub-skills, not just consumption of the ecosystem.

We survived a genuine, undocumented regional cloud outage mid-build and came out the other side with a cleaner, more resilient deployment than we started with.

We found and fixed three real, previously-undocumented DataHub server-side validation quirks that the SDK's own type hints actively mislead you about — and documented every one, publicly, in the README, instead of quietly papering over them.

Every one of the six proof-of-real-work features leaves a permanent, independently checkable trace — a real Assertion, a real fixed ownership gap, a real recalled playbook, a real warning on a real ML model's DataHub page — not just a line in RippleX's own dashboard.

The live activity feed is genuinely live, not a fake progress bar: it polls the same audit trail that's being written to Postgres in real time, the instant each agent step happens.

What We Learned

The gap between "the type hints say this is allowed" and "the server will actually accept it" is where the real bugs live. Every validation error we hit came from that gap, not from our own logic — a reminder that live verification against a real service is not optional for anything that writes to an external system.

DataHub's high-level convenience SDK wrappers and the underlying OSS aspect classes are two genuinely different products wearing the same name. Knowing which one you're actually calling — and which one is Cloud-gated — matters, and it's only discoverable by introspecting the installed SDK, not by reading marketing copy.

Infrastructure has its own agency. A regional capacity shortage doesn't know or care about a deadline, and the only real defense is being able to move fast when it happens, not just wait it out.

The most convincing way to prove an agent did real work isn't a longer demo video — it's a permanent object a judge can go find themselves, days later, still sitting inside DataHub. That single idea shaped every feature built after the core pipeline worked.

What's Next for RippleX

Fund real LLM billing so the reasoning steps run on genuine Claude output in production by default, with the deterministic path staying as the safety net it was always meant to be.

A second, differently-typed planted column in the demo estate, so Compounding Memory's "sibling migration" demo spans two genuinely different tables instead of a repeat run on the same column.

Get DataHub Skills PR #59 merged upstream — currently open, not yet in the shared skill library.

Extend ML Blast-Radius Guard one hop further, from MLModel into MLModelDeployment, to flag the actual serving endpoint at risk, not just the model artifact.

Build out the GKE Helm chart (currently deliberately scaffolded but unbuilt) for teams that already run DataHub on Kubernetes and want RippleX to live there too.

By the Numbers (from real, live verification runs)

  • 21 real downstream consumers found via real DataHub lineage traversal on a single planted migration.
  • 6 real, ordered GitHub pull requests opened per run, against a genuinely separate consumer repository.
  • 100% of chained parallel-run verifications passed, row-for-row, against the real warehouse.
  • 18 + 18 real ownership and description gaps found and fixed by Metadata Debt Radar in a single run.
  • 6/6 real DataHub Assertions created and confirmed readable back from DataHub with status: COMPLETE.
  • 1 real production ML model found and flagged via a genuine second lineage hop.
  • 3 real, previously-undocumented DataHub server-side validation bugs found and fixed, live.
  • 4 us-central1 zones exhausted, 2 machine types tried, 1 genuine region migration, 0 data lost.

Built With

Share this project:

Updates