Blastradar

Know what ML breaks before you merge. Blastradar traces a data PR's changed columns through DataHub lineage to the ML models they'd silently break, and comments on the PR before it lands.

Comment

Inspiration

Dropping a column from a data model doesn't throw an error. Nothing turns red. The feature pipeline just starts emitting nulls, and a production ML model quietly degrades for weeks before anyone connects the drop in accuracy back to a merged PR.

The data engineer who made the change had no way to know an ML system was downstream. The ML team had no way to know the change was coming. Both were doing their jobs correctly — the information simply didn't exist at the moment the decision was made. We wanted to put it there: at review time, in the PR, before the merge.

What it does

Blastradar is a CI agent that reviews data pull requests for downstream machine-learning impact.

When someone opens a PR changing a SQL/dbt model, it:

  1. Parses the diff with sqlglot to determine exactly which columns changed — drops, renames, type changes — resolving CTEs, aliases, and SELECT *
  2. Resolves those columns to DataHub URNs
  3. Walks DataHub's column-level lineage downstream — a hop-capped breadth-first traversal with a cycle guard — until it reaches ML entities (mlFeatureTable, mlModel, mlModelDeployment), preserving every distinct path
  4. Determines whether each model was trained on the changed column or merely reads it at inference, using dataProcessInstance training-run provenance
  5. Scores severity deterministically, then posts a plain-English PR comment
  6. Writes the finding back into DataHub as an incident, a tag, and a saved document

The trained-on vs. inference-only distinction is the part we care most about. A model that was trained on a column doesn't just read a null — its learned relationship to that feature is invalidated. That's a materially different problem from an inference-time null, and it's the difference between "monitor this" and "retrain this."

How we built it

The core architectural decision: the impact determination is fully deterministic; the LLM only writes prose.

Lineage traversal, ML-terminal detection, training-set provenance, and severity scoring are plain Python running a fixed algorithm against DataHub. Exactly one LLM call happens, at the very end, given an already-resolved impact graph, asked only to write the explanation and suggest a migration. It never decides what is impacted or re-ranks anything.

We did this because judges and users run a tool once. An agent looping over graph traversal produces a different answer each time, and a non-deterministic demo reads as broken. Determinism is also what makes the output auditable — every severity carries the specific rule clauses that fired.

The narrator supports Groq (default) and Anthropic, auto-selected by whichever API key is present, with a fully templated fallback if neither is set — so the tool produces a complete, useful comment with no API key at all.

Everything runs on DataHub Core (open source) — no Cloud-only features on the critical path. Write-back uses incidents, tags, and documents only.

We also built two reproduction paths: make demo runs the entire pipeline against recorded fixtures with no DataHub instance, no network, and no API key, in under a second; make demo-live runs against a real local DataHub. The fixtures double as the test suite.

Challenges we ran into

None of DataHub's sample datasets ship ML entities. There was no feature-table → model → deployment chain to traverse, so before writing a line of agent logic we had to seed one — model groups, versions, feature tables with features sourced from real dataset columns, deployments, and training-run entities with real input datasets. We packaged that work as a reusable datapack and contributed it upstream.

The Python SDK has changed significantly across versions, and model training data blends generations of it. We stopped writing DataHub calls from memory entirely and introspected the installed package for every signature, cross-checking against live docs, recording what we verified in docs/API-NOTES.md. That single discipline saved the most time.

A demo that needs infrastructure is a demo nobody runs. Getting to a genuinely offline make demo meant designing the DataHub client for record/replay from the start and selecting replayed responses by request signature rather than call order, so the fixtures wouldn't be brittle.

Late in the build we hit a DataHub datapack-loader bug (KeyError: 'Did not find a registered class for c', datahub#11107) that blocked loading a richer base graph locally. Rather than ship something unverified against the deadline, we scoped the demo graph honestly and documented exactly what the engine handles versus what the bundled graph exercises.

Accomplishments that we're proud of

  • It runs in under a second, offline, with no setup. make demo needs no DataHub, no network, no API key.
  • The trained-on vs. inference-only distinction — we haven't seen another tool make it, and it's the difference between monitoring and retraining.
  • 99 passing tests, all offline, covering the traversal, the scoring rules, the CLI, and both narration providers.
  • Two upstream contributions to DataHub — a datahub-ml-impact skill (PR #78) and an ml-showcase datapack (PR #18813) filling the ML-entity gap we hit ourselves.
  • A limitations section we're willing to stand behind — including which capabilities are test-verified and which aren't.

What we learned

Metadata is only valuable at the moment a decision is made. DataHub knew the entire lineage chain the whole time; the problem was never missing data, it was that nobody queries a catalog while reviewing a diff. Moving existing knowledge to the point of decision turned out to be far more valuable than generating new knowledge.

We also learned to be deliberate about where the LLM sits. The instinct is to let an agent explore the graph. The better design gives it a solved problem and asks only for language — narrower, faster, cheaper, reproducible, and auditable.

And: verify the API against the installed package, not against memory.

What's next for Blastradar

  • Deeper demo graph — the engine handles multi-hop chains and cycles (test-verified); we want the bundled graph to dramatize them once the loader bug is resolved
  • Richer PR-time signals — per-hop SQL transformations where DataHub exposes query text
  • Retraining suggestions, not just migration hints, for trained-on impacts
  • Landing the upstream PRs so the ML-entity gap is closed for everyone
  • Beyond dbt — the diff analyzer is sqlglot-based and dialect-configurable, so Airflow and Spark pipelines are a natural extension

Built With

Share this project:

Updates