Dataripple

• 

  Dataripple (demo-ui)

Inspiration

Database schema changes can look harmless in a pull request but still break systems that nobody knew depended on them. A developer might drop or rename a column, see all application tests pass, and merge the change—only to discover later that a downstream service, analytics pipeline, or dashboard was quietly relying on that data.

We were inspired by a simple question: What if we could detect the dependencies that our metadata and lineage don't know about before a schema change reaches production?

DataHub already provides valuable organizational context through metadata and lineage. We wanted to build on top of that foundation by combining declared lineage with observed usage evidence, so teams could make safer schema-change decisions before merging a PR.

What it does

DataRipple is a shift-left schema-change impact analysis agent that runs as part of the GitHub pull request workflow.

When a developer proposes a database schema change, DataRipple:

1. Parses SQL DDL and dbt schema changes.
2. Retrieves relevant context from DataHub, including lineage and governance metadata.
3. Reconciles declared lineage with observed query usage to identify potential hidden or undeclared consumers.
4. Evaluates metadata coverage and evidence confidence.
5. Calculates a deterministic risk level: LOW, MEDIUM, HIGH, or CRITICAL.
6. Identifies safer migration patterns, such as Expand-Contract for potentially breaking renames.
7. Generates an evidence-backed impact report for the pull request.
8. Records migration analysis and decisions back into DataHub where appropriate.

The core idea is simple:

Declared lineage + observed runtime evidence → actual dependency context → schema-change blast radius.

Instead of discovering the impact of a schema change after production breaks, DataRipple helps teams discover it while the change is still a pull request.

How we built it

DataRipple is built as a modular TypeScript monorepo using pnpm workspaces.

The system is organized into independent packages for schema-diff parsing, DataHub integration, lineage resolution, usage reconciliation, metadata coverage, deterministic risk classification, migration-pattern detection, GitHub integration, and PR reporting.

The analysis pipeline follows this flow:

GitHub PR → Schema Diff → DataHub Context Retrieval → Evidence Reconciliation → Coverage Assessment → Risk Classification → Migration Strategy → PR Report → DataHub Write-back

We use DataHub as the organizational context layer, accessing relevant metadata through its integration interfaces and using that context alongside observed usage evidence to assess the real blast radius of a schema change.

We deliberately kept risk classification deterministic and auditable rather than relying on an LLM to make critical decisions. AI and agentic workflows can assist with context discovery, explanations, and migration recommendations, while the final risk decision is based on explicit evidence and rules.

The project also includes an interactive visual demo UI that demonstrates destructive drops, column renames, safe additive changes, hidden consumer discovery, risk classification, and migration recommendations.

Challenges we ran into

One of the biggest challenges was reasoning about the difference between declared dependencies and actual dependencies.

A lineage graph can be correct according to everything an organization has registered, while still being incomplete because services, scripts, or other consumers may access data without being represented in the lineage.

We also had to design a risk engine that could distinguish between different types of schema changes and avoid treating every change as equally dangerous. A column addition, a rename, a type change, and a destructive drop can have completely different consequences depending on downstream usage.

Another challenge was detecting semantic renames instead of incorrectly treating them as an unrelated column drop and addition. We implemented deterministic heuristics to improve rename detection and connect those changes to safer migration strategies.

Finally, we had to design the system so that risk decisions were explainable. A result such as "CRITICAL" is not useful by itself, so DataRipple focuses on presenting the evidence behind the decision—what changed, which consumers are affected, what usage was observed, and why the change was classified as risky.

Accomplishments that we're proud of

We are proud of building a complete end-to-end workflow rather than just a schema-diff parser or a lineage viewer.

DataRipple connects the entire path from a GitHub pull request to schema analysis, DataHub context retrieval, evidence reconciliation, deterministic risk classification, actionable migration guidance, and persistent reporting.

We are particularly proud of the Declared vs. Observed approach. The system is designed to identify situations where the dependency graph represented by organizational metadata does not match what systems are actually doing.

We also built the project as a modular TypeScript monorepo with a comprehensive automated test suite covering the core parsing, reconciliation, and risk-analysis logic.

Most importantly, we are proud of the idea that DataRipple can help turn DataHub from passive organizational metadata into an active part of the schema-change decision process.

What we learned

We learned that schema safety is not just about understanding what changed—it is about understanding who or what depends on what changed.

We also learned that metadata quality and runtime evidence complement each other. Declared lineage provides valuable organizational context, while observed usage can reveal dependencies that have not yet been documented.

Another important lesson was that systems making engineering decisions need to be explainable. A deterministic, evidence-backed risk engine makes it easier for engineers to understand, trust, and challenge the result.

Finally, we learned that building on top of an existing metadata platform like DataHub can be much more powerful than building another isolated tool. The real value comes from connecting new operational signals to existing organizational knowledge and feeding useful discoveries back into that ecosystem.

What's next for DataRipple

The next step is to make DataRipple easier for teams to adopt directly in their development workflows through a zero-configuration CLI and GitHub Action.

We also want to expand support for more schema-change sources and migration frameworks, improve runtime dependency discovery, and strengthen the connection between observed usage evidence and DataHub's context graph.

Longer term, we envision DataRipple becoming a continuous feedback loop:

DataHub provides organizational context → DataRipple observes real-world dependency behavior → schema changes are analyzed before deployment → newly discovered knowledge is written back into DataHub.

Our goal is to help data and platform teams move from simply documenting dependencies to continuously validating them against reality—so the next schema change is safer than the last one.

Built With

• api
• datahub
• dbt
• graph
• mcp
• monorepo
• next.js
• node.js
• pnpm
• postgresql
• react
• rest
• sql
• typescript