Blast Radar

Inspiration

Every data team has the same fifteen minutes before a migration: someone is dropping a column, the lineage tab is open, and twelve downstream assets are listed. The information needed to answer "who does this hurt" is right there, and it still does not answer the question, because a lineage graph tells you what is downstream, not what matters downstream. Two of those twelve are a director's dashboard and a churn model. Nothing on the screen says which two.

So the change ships, and the answer arrives on Monday as a Slack message from someone whose report is empty.

What it does

Point it at the asset about to change:

blast-radar scan "urn:li:dataset:(...analytics.order_details,PROD)" \
    --change "dropping column promo_code" --publish

It walks the downstream cone through DataHub's MCP server, scores every consumer, and produces a ranked impact brief: what breaks first, who to tell, and which impacted assets have no owner at all. With --publish the verdict goes back into DataHub as metadata: a severity tag, a numeric risk score in a structured property, and the brief itself saved as a document linked to the changed asset and to everything it endangers.

Against DataHub's own showcase-ecommerce catalog, one dbt model produced 35 downstream assets, 16 of them high severity, and 21 with nobody to notify.

The one design decision worth defending

The score is not generated by a language model. It is a weighted sum of six normalised factors, each carrying the evidence that produced it.

A ranking that changes between two runs cannot sit in a release gate. If a data engineer is going to delay a migration because of a number, that number has to be reproducible, auditable, and arguable. So the model is deterministic, and identical inputs give an identical ranking every time.

How we built it

Python 3.11, the official MCP Python SDK as a client, and mcp-server-datahub launched over stdio. Scanning talks to DataHub exclusively through MCP: get_lineage, get_entities and get_dataset_queries going in, and search_documents, remove_tags, add_tags, add_structured_properties and save_document coming back out.

The layers are split so that most of the system is testable without a catalog: the graph walk takes a fetch callable rather than an MCP session, the scoring model takes plain data, and the write-back policy returns intents rather than tool calls. The whole suite runs with no DataHub instance and no network, including an end-to-end scan against an in-memory catalog.

Challenges we ran into

Proximity was measuring the wrong thing. The first live run ranked intermediate staging tables above the charts an executive opens. The dashboard was five edges from the change and scored below a staging table one edge away. But the last two of those five edges were a chart rendering a dataset and a dashboard framing that chart, and neither reshapes data, so neither can absorb a dropped column. Distance now counts transforms, not edges, and the brief says both: "5 hops downstream of the change (3 transforms deep)".

DataHub rejects a structured property it has never been told about. Writing a value to an undefined property soft-defines it, scoped to whichever entity type happened to be written first. Scores landed on datasets and every chart was refused with "no valid property assignments remain after removing values for non-existent properties". Tags fail the same way for a different reason: "Failed to validate label ... Urn does not exist". Both now happen once, up front, in a setup command.

Two DataHub responses, neither a superset of the other. Some assets' tags appear only on the lineage hit; full ownership appears only on the entity fetch. The first version overwrote one with the other and silently dropped whichever signal the winner lacked. Enrichment merges field by field now.

Catalog metadata is untrusted input. Asset names come from whoever has ingestion rights, and they end up in a brief published back into the graph. An asset named orders[/bold] crashed the terminal render after a scan had already written its verdict; one named orders | 99 | LOW forged a row in the published brief, complete with a score nobody computed. Names are escaped per renderer now, and characters that carry no glyph are stripped where the metadata enters.

Accomplishments that we're proud of

Verified in DataHub rather than trusting the tool's own success response: sixteen assets carrying a severity tag, the charts carrying their scores, and the brief queryable as a document.

And the ranking finally reads the way a data engineer would: named charts, "Executive Summary" and "Customer Analysis", above the intermediate measures tables that sit closer to the change.

What we learned

Tool schemas describe arguments and say nothing about result shapes, so every mapping in this project was written against real responses from a live v1.6.0 instance. The habit that paid off most was refusing to write a single line against a guessed payload: connect first, dump what comes back, then write the code.

The second lesson was about tests that cannot fail. A characterization suite pins the exact published numbers, because the behavioural tests only ever compared scores to each other, and would have passed just as happily if the whole scale had shifted.

What's next for Blast Radar

Column-level blast radius using get_lineage_paths_between, so the score reflects whether the specific column being dropped actually reaches a given dashboard. A CI mode that fails a pull request when a migration touches a critical consumer without an owner. And notifying owners where they already are, instead of leaving the brief in the catalog for someone to find.

Built With

• datahub
• mcp
• pytest
• python
• rich
• typer
• uv