blast radius

Blast Radius automatically analyzes DataHub lineage, fixes downstream schema changes, and opens ready-to-review pull requests.

Comment

Blast Radius: schema-change auto-remediation for DataHub

Inspiration

Every data team has lived this moment: someone wants to rename a column, and nobody actually knows what breaks. A single column like users.email might silently feed a dbt model, an Airflow pipeline, an ML feature table, and an executive dashboard. Today, finding that out means an engineer manually grepping across repos, half-remembering which ones matter, and still missing something. DataHub already maps all of this in its lineage graph, but knowing what's downstream and actually fixing it are two very different problems. We wanted to close that gap by not just telling you what breaks, but fixing it and proving the fix works.

What it does

Give Blast Radius a table, a column, and a new name, and it:

  1. Reads DataHub's real lineage graph to find everything downstream of that column.
  2. Auto-generates a working SQL fix for every dbt-style model that references it using a real SQL AST rewrite instead of regex, correctly handling dbt's Jinja templating.
  3. Generates a suggested fix for riskier Python and Airflow code. These are flagged for human review rather than auto-merged because an LLM rewrite of arbitrary Python is far less predictable than a parsed SQL rename.
  4. Flags anything with no source file to edit, such as a dashboard, for manual review.
  5. Opens a real, clearly labeled GitHub pull request with everything organized by confidence.
  6. Writes a record back onto the DataHub entity itself, allowing the next person who looks at that column to inherit the history, not just the current state.

The scope is deliberately narrow. Rename-column only. One action done correctly beats several done shakily.

It works in three ways: a clean web UI with dropdowns (Command Mode), a natural-language chat interface powered by Groq (Chat Mode), and a terminal CLI. All three call the exact same underlying pipeline, so there is no duplicated logic between them.

How we built it

  • Backend: Python + FastAPI, structured around one shared analyze() and apply_fix() pipeline that every interface calls into.
  • DataHub integration: Direct GraphQL queries against DataHub's GMS API covering dataset-level lineage, dataJob input and output relationships, and chart upstream lineage. These are merged and deduplicated into a single dependency graph.
  • SQL rewriting: sqlglot parses each SQL file into a real AST, renames matching column references, and re-emits the SQL with a Jinja protection layer so dbt's {{ ref(...) }} templating survives the round-trip untouched.
  • Suggested fixes: Groq (Llama 3.3 70B) generates Python and YAML rewrites, kept clearly separate from the auto-applied SQL fixes.
  • Chat Mode: Groq also acts as a thin translator, converting a plain-English request into the same structured {table, column, new_name} action that Command Mode's dropdowns produce. There is no separate logic path.
  • PR automation: PyGithub opens a real pull request with generated diffs and a structured description explaining what was auto-applied, what is suggested, and what is flagged.
  • Write-back: An institutional-memory note is emitted back onto the DataHub column through a MetadataChangeProposal, recording what was analyzed and which PR resulted.
  • Frontend: React + Vite, with a custom blast radius radar visualization showing the target column at the center and downstream dependencies radiating outward, color coded by confidence.

Challenges we ran into

Nearly every integration point required correcting a wrong assumption about DataHub's real GraphQL schema rather than the one we expected.

  • A SchemaFieldDataType field we assumed was an object turned out to be a leaf type, so no subselection was allowed.
  • Column-level (fine-grained) lineage through a SchemaFieldEntity query did not behave the way the documentation implied. We fell back to dataset-level lineage instead.
  • A guessed "Consumes" relationship type did not exist in this DataHub version. We had to introspect the live GraphQL schema directly using __type queries to find the actual field names and relationship shapes for dataJob and chart entities.
  • Even after finding the right queries, a subtle logic bug compared the wrong URN at each hop of the dependency chain. The fix required tracing each entity's real relationship one hop at a time, from users to user_contact_index to fetch_contactable_users, rather than jumping directly from users to fetch_contactable_users.
  • ChartInfoClass's lastModified field turned out to be non-nullable, which broke metadata emission with an opaque Avro serialization error. The root cause was a missing ChangeAuditStampsClass.
  • The Groq client was being constructed at import time, causing the application to crash on startup whenever the API key had not yet been configured. We fixed this with lazy initialization.

Getting from mocked code with passing tests to a system that genuinely worked against a live DataHub instance required real, iterative debugging. We repeatedly verified assumptions against the actual GraphQL schema instead of guessing.

Accomplishments that we're proud of

  • A fully live, end-to-end run using real DataHub lineage, real classification and confidence scoring, real SQL AST rewriting, a real GitHub pull request, and real write-back to DataHub. No mocks exist anywhere in the final pipeline.
  • Detecting dependencies across three distinct DataHub entity types: dataset, dataJob, and chart, within a single analysis rather than only simple dataset-to-dataset lineage.
  • A SQL rewriter that correctly preserves real dbt Jinja templating instead of working only on plain SQL.
  • Three genuinely interchangeable interfaces: Command Mode, Chat Mode, and a CLI, all sharing one pipeline with zero duplicated logic.
  • An examples/ folder containing real, live-generated output rather than synthetic placeholders, allowing the fix quality to be inspected without running anything.

What we learned

  • DataHub's public GraphQL schema documentation does not always match what a deployed version actually supports. Introspecting the live schema directly with __type queries proved far more reliable than relying on assumptions.
  • Precision matters more than coverage. Narrowing the scope to "rename column, done correctly" produced a far more trustworthy tool than trying to support delete operations and type changes at the same time.
  • Automatically applying fixes and merely suggesting them represent different trust levels and deserve separate code paths. SQL's structured, AST-parseable nature makes automatic application appropriate, while Python's flexibility does not.
  • Writing changes back into the same system you read from is what truly closes the loop. Most lineage tools stop at visualization.

What's next for Blast Radius

  • Extend detection to ML feature tables with real structural lineage edges. The current demo seed data only contains descriptive notes rather than queryable relationships.
  • Support additional schema changes beyond renames, including type changes and safe column deletion, each with its own confidence model.
  • Expand the Python and YAML suggestion path with static analysis to increase confidence for simpler cases.
  • Add multi-hop impact analysis that traces dependencies of dependencies rather than stopping after one downstream level.
  • Contribute improvements back to DataHub itself based on what we learned while working with its GraphQL lineage API.

Built With

Share this project:

Updates