Blast Radius

Catch breaking changes before they break production

Comment

Inspiration

Anyone who's worked near a production ML pipeline knows this feeling: something is subtly wrong with the model's output, and nobody can say when it started. Somebody renamed a column three weeks ago. The pipeline never crashed. The model never threw an error. It just quietly started serving worse predictions, and by the time anyone noticed, the damage was already downstream of the damage.

That's the failure mode I wanted to attack. Not the loud failures — those get caught. The silent ones.

The insight that made it feel solvable: the information needed to catch this already exists in DataHub. The lineage graph knows exactly what depends on what. The problem isn't missing data, it's that nobody's watching the graph at the moment a change happens. So I built something that does.

What it does

BlastRadius is an agent that answers one question automatically: someone just changed this — what's about to break?

Given a changed entity and a description of the change, it:

  1. Traverses DataHub's lineage graph to find every downstream asset at risk
  2. Reasons about severity — is this genuinely breaking, or harmless?
  3. Writes the verdict back into DataHub: a severity tag plus a plain-English warning appended to each affected entity's description

Point 3 is the part I care most about. Producing a report is easy, but reports get closed and forgotten. Writing the warning onto the entity itself means the next engineer who opens that model in DataHub sees it — in the place they were already going to look.

Demo scenario: a three-stage pipeline, raw_user_events → user_feature_table → recommender_model_v1. Drop the purchase_category column from the raw table and the agent traces the chain, correctly identifies it as breaking (that column feeds the category_affinity feature, which the model ranks on), and tags both downstream entities.

How we built it

Stack: DataHub MCP Server (mcp-server-datahub), Groq for inference (Llama 3.3 70B, free tier), Python.

The architecture decision that mattered most was making this a real agent rather than a script with an LLM stapled to the end. My first version called get_lineage in Python, passed the result to the model, and asked for a verdict. It worked, but the LLM wasn't doing anything an if statement couldn't.

The rebuild inverted that. The LLM gets four DataHub MCP tools — get_lineage, search, add_tags, update_description — and drives them itself in a tool-calling loop. It decides when to traverse deeper, which entities to tag, what to write. The Python code is plumbing: a dispatcher that routes tool calls to the real MCP functions and hands results back.

Everything runs locally against DataHub quickstart, on Groq's free tier. Total cost to run: nothing.

Challenges we ran into

The agent hallucinated URNs. This was the interesting one. DataHub quickstart ships with sample data, and the model started confidently tagging entities like user_purchase_insights that were nowhere in my pipeline — it had seen similar names in search results and pattern-matched its way to plausible-sounding URNs that didn't exist. Every write failed with "Entity does not exist." The fix was constraining it explicitly in both the system prompt and the tool descriptions: only operate on URNs that get_lineage actually returned, never construct one. A good reminder that a model being fluent about your data isn't the same as it being correct about your data.

Tags have to exist before you can apply them. batchAddTags kept failing until I understood that in DataHub a tag is itself an entity — you can't apply blastradius-breaking until you've created it. Added a bootstrap step that emits all three severity tags on startup.

Groq rejected my message history. The SDK attaches fields to assistant messages (annotations, executed_tools) that the API then refuses on the way back in. Naively round-tripping model_dump() breaks the loop on turn two. Now I construct the assistant message explicitly with only the fields the API accepts.

And the unglamorous ones: the machine ran out of disk entirely mid-build, Docker Desktop refused to install on the Windows version I had, and my venv was silently pinned to Python 3.10 while mcp-server-datahub requires 3.11+. Roughly half the total time went to environment problems rather than code.

Accomplishments that we're proud of

The agent genuinely reasons rather than following a script. Watching it decide on its own to traverse a second hop, then tag both entities it found, then write a specific explanation naming the exact feature at risk — that behavior wasn't coded, it emerged from the tools and the prompt.

It closes the loop back into DataHub. Reading metadata is table stakes; contributing back to the graph is what makes it useful to a team rather than a demo.

38 unit tests, all fully mocked — they run in under a second with no Docker, no DataHub, no API key. I wrote them before trusting the thing against a live instance, and they caught three real bugs in the dispatch layer.

Also, honestly: getting from zero knowledge of DataHub, MCP, and agentic loops to a working system, solo, in a handful of days.

What we learned

The line between "LLM feature" and "agent" is whether the model holds the tools. Once it drives the calls, you stop writing control flow and start writing constraints — and the constraints become the hard part.

Grounding matters more than capability. The hallucinated-URN bug wasn't a weak model, it was a model given room to guess. Removing that room fixed it.

Write-back is the difference between insight and impact. A warning nobody sees is worth nothing.

Mock your tests. Being able to validate logic in 0.6 seconds without spinning up seven Docker containers changed how fast I could iterate.

What's next for Blast Radius

Column-level lineage. Right now impact is assessed at table granularity. DataHub supports column-level lineage, which would let the agent say "this specific field breaks" rather than "this table is affected."

Automatic triggers. The change description is currently passed by hand. In production this should fire off a schema-change event or a CI hook, so the warning lands before the PR merges rather than after.

Native ML entity types. The demo models everything as datasets. Using DataHub's mlFeatureTable and mlModel entities would map more precisely onto real ML lineage.

Historical blast radius. Track predicted impact against what actually broke, and you'd have a feedback signal for calibrating the severity judgments over time.

Built With

  • agentic-ai
  • ai-agent
  • data-engineering
  • data-governance
  • data-lineage
  • datahub
  • docker
  • graphql
  • groq
  • llama
  • llm
  • machine-learning
  • mcp
  • metadata
  • mlops
  • model-context-protocol
  • production-ml
  • pytest
  • python
  • tool-calling
Share this project:

Updates