Inspiration
I renamed a column once. Small migration, clean diff, sailed through review. Three days later a dashboard was showing zeros and it took most of an afternoon to work out why: my rename, six steps downstream, in a chart nobody thought to connect back to that table.
The frustrating part is the answer was never actually missing. DataHub already knew that dashboard depended on that column. It just wasn't in front of anyone at the one moment it would have mattered, which is while the pull request was still open. I wanted that answer to show up as a comment on the PR itself, before the merge, so nobody has to lose an afternoon finding it the way I did.
What it does
Breaking-Change Radar runs as a GitHub Action the moment someone proposes a schema change.
It reads the diff and pulls out the table, the column, and what kind of change it is. Then it queries DataHub for the blast radius, combining column-level and dataset-level lineage so a dashboard or ML model hanging off the parent table doesn't get missed. Anything that genuinely has nothing to do with the changed column gets pruned back out. Every impacted asset gets scored, a verdict gets issued, a migration plan comes out grouped by owning team, and the whole thing lands as a pull request comment.
Then it writes back into DataHub itself: a saved report on the asset, incidents on the high-severity ones so they show unhealthy, structured properties carrying the severity and a link back to the report, and a tag. There's a cleanup command that undoes all four.
In the demo, renaming raw.customers.customer_id turns out to impact 16 downstream assets across 5 platforms and 5 lineage hops, owned by 5 different teams. That set includes a SageMaker model sitting five hops from the column that changed.
How I built it
Python, built around what I ended up thinking of as agentic bookends. A language model up front, reading unstructured input. Deterministic Python in the middle, doing the actual investigation. A language model again at the back, writing the remediation prose. Then deterministic Python one more time, checking and rendering whatever the model wrote.
Lineage and metadata come from the DataHub MCP Server, with the GraphQL API filling in directly for incidents, document deletion, and a lineage fallback that turned out to be load-bearing on a freshly seeded instance. Reports run on Groq, with Google Gemini as failover, and a deterministic template underneath both of those, so the tool still works even with no API key at all.
The rule I kept coming back to: the model is allowed to improve the plan, but it never gets to decide whether something is actually broken. Every fact that matters is computed in Python, straight from what DataHub itself reports.
The DataHub Python SDK handled the parts around the edges: seeding the demo graph, and the hard deletes during cleanup.
Challenges I ran into
Honestly, most of the real work was finding the places where the tool was confidently wrong.
An empty lineage response was being read as "no impact." It looked like an all-clear, and it was actually a failure to even look. That's now an explicit UNABLE TO ASSESS state. The trade-off: a genuine leaf table with nothing downstream now says it can't assess rather than claiming it's safe.
At one point the migration plan recommended doing the rename first and migrating consumers afterward, which is exactly the outage this whole project exists to prevent. The transition order is now fixed per change type in code and handed to the model as a given rather than left for it to decide.
The part of the pull request comment holding the migration plan was being located by matching a literal string against the model's own output. The model shifted its heading wording slightly between runs, and a correct five-owner plan got silently replaced by a hardcoded fallback sentence, and the log still reported success.
The impacted-assets table itself was being written by the model, and it drifted: right counts, wrong platform names, dashboard names that didn't exist. Python now renders that table itself and splices it over whatever the model produced. Two days before submission I found that splice was only firing on some runs, not all of them, and had to fix that too.
Every single one of these looked like a healthy run while it was happening. That pattern is basically what the whole project ended up organized around catching.
Accomplishments I'm proud of
That it admits what it doesn't know. Four of the sixteen findings in the demo are confirmed by real column-level lineage, and twelve are inferred from dataset-level lineage. It says which is which, rather than presenting all sixteen as equally certain.
The verdict has four possible values, and SAFE is only reachable through exactly one rule. Any state nobody anticipated falls through to UNABLE TO ASSESS instead of quietly showing a green badge.
It's held up against live pull requests, not only dry runs. The comment updates in place across consecutive runs instead of piling up, and all four change types have been independently verified end to end.
And it runs with zero API keys at all, in a mode that says plainly what it isn't doing, instead of pretending everything's normal.
What I learned
That graceful degradation is the real enemy. Nearly every defect in this project was a fallback that fired quietly and let a broken run look perfectly healthy. A fallback that doesn't announce itself is a defect in its own right.
Also: a validator that checks generated output against computed facts can't catch a fault sitting in the computation itself. It passes cleanly on wrong input, every time.
What's next for Breaking-Change Radar
Actually generating the compatibility artifact instead of only describing it, then re-running the impact analysis against freshly re-queried lineage to show the risk genuinely dropping. The line I'd hold myself to: that re-check has to be driven by real lineage data. A run that reports SAFE just because it was told the fix went in would be theatre, not verification.
Beyond that, calibrating the severity model against a real warehouse instead of a 34-entity synthetic graph, and covering more change types than the four it handles today.
Log in or sign up for Devpost to join the conversation.