ShadowGraph

A DataHub-powered pre-merge safety agent that traces downstream lineage, replays SQL changes, and blocks pull requests that silently break critical metrics or ML features.

Comment

Inspiration

Traditional data CI is good at catching structural problems such as dropped columns, incompatible types, and invalid SQL. It often misses semantic changes where a field keeps the same name and type but changes meaning.

Our reference example uses discount_percentage. Originally, 25 means 25%, so the transformation divides it by 100. A pull request removes that conversion and treats 25 as a fraction instead. The schema remains valid and the SQL compiles, but downstream revenue becomes negative and an ML feature leaves its expected range.

ShadowGraph was inspired by the idea that a pull request should be evaluated using the organization’s real data graph before it reaches production.

What it does

ShadowGraph is a DataHub-powered pre-merge safety agent for data changes.

It follows an evidence-backed workflow:

DETECT → RESOLVE → TRACE → REPLAY → COMPARE → DECIDE → RECORD

ShadowGraph:

  1. Reads the immutable base and head commits of a pull request.
  2. Detects changes in SQL, dbt models, schemas, columns, and expressions.
  3. Resolves affected datasets and columns to canonical DataHub URNs.
  4. Traverses bounded downstream DataHub lineage.
  5. Identifies true column consumers while excluding false positives.
  6. Replays affected before-and-after transformations in DuckDB.
  7. Compares schemas, row counts, null rates, metrics, and distributions.
  8. Produces a deterministic success, failure, or inconclusive decision.
  9. Publishes a commit-scoped GitHub Check with owners and supporting evidence.
  10. Prepares an approval-gated evidence record for DataHub.

The dashboard can load a real GitHub pull request and display its SQL diff, commit identity, DataHub context, downstream consumers, owners, replay measurements, and merge decision.

How we built it

ShadowGraph is implemented primarily in TypeScript and React using vinext, Vite, and the Cloudflare Workers-compatible runtime.

DataHub is the context and metadata layer. ShadowGraph uses DataHub for:

  • Canonical dataset and column identity
  • Schema metadata
  • Table-level and column-level lineage
  • Downstream dependency discovery
  • Asset ownership
  • Governance and review-routing context

We integrated the official DataHub MCP Server and DataHub Skills in addition to a typed GraphQL adapter.

For behavioral validation, ShadowGraph compiles the affected dbt-style dependency subgraph and executes the before-and-after transformations in DuckDB. Replay is bounded and read-only. It measures business metrics and distributions without exposing or logging complete row data.

GitHub Actions analyzes the exact pull-request base and head SHAs. Each run produces a versioned evidence artifact containing lineage, consumers, owners, replay measurements, breached thresholds, and the final decision.

The ShadowGraph UI downloads this evidence server-side and validates it against the current repository, pull request, base SHA, and head SHA. Evidence from another commit is rejected.

The merge decision is deterministic. An optional local Ollama model can summarize existing evidence, but it cannot change whether a pull request passes or fails. No paid LLM API is required.

Challenges we ran into

One major challenge was reliable DataHub lineage traversal. A single deep lineage query could time out even when a one-hop query returned quickly. We implemented bounded one-hop breadth-first traversal with:

  • Cycle prevention
  • A visited set
  • Node and edge deduplication
  • Hard depth and result limits
  • Real parent-to-child edges

We also found a subtle dataset URN parsing bug. A DataHub dataset URN contains the platform, qualified dataset name, and environment. Early parsing incorrectly displayed PROD as the dataset name. Regression tests now ensure the final dataset component, such as stg_orders, is displayed correctly.

Another challenge was preventing false confidence. The first UI used deterministic demonstration data, but a live failure could not be allowed to silently fall back to that scenario. We introduced explicit source labels:

  • LIVE GITHUB
  • LIVE DATAHUB
  • COMMIT-SCOPED EVIDENCE
  • DEMO DATA
  • UNAVAILABLE

Transporting detailed evidence through GitHub was also challenging. GitHub Check prose was insufficient for rendering real measurements and lineage. We added a versioned JSON workflow artifact and safe server-side artifact retrieval, including immutable SHA validation, archive bounds, authentication handling, and stale-evidence rejection.

Finally, local vinext routes run inside Miniflare, which does not inherit ordinary shell environment variables. We documented the secure .dev.vars mechanism for providing the server-side GitHub token while ensuring it remains ignored by Git and absent from API responses.

Accomplishments that we're proud of

We tested ShadowGraph using two real GitHub pull requests.

The dangerous pull request keeps the same schema but removes the percentage normalization. ShadowGraph identifies two true downstream consumers:

  • fct_order_revenue
  • order_discount_features

It detects eight behavioral breaches, including:

Measurement Before After
Total net revenue 17,311.4075 -445,403.95
Average discount rate 0.149 14.9
Maximum discount rate 0.35 35
High-discount orders 6 17

ShadowGraph publishes a failed GitHub Check titled Unsafe data change blocked, assigns critical risk, and routes the result to Analytics Engineering.

Our second pull request restructures the same transformation using a CTE without changing its behavior. It reaches the same downstream consumers but produces zero breaches, so the Check succeeds. This proves ShadowGraph can block dangerous semantic changes without blocking safe refactors.

We are also proud that:

  • DataHub is foundational rather than decorative.
  • Every result is bound to an immutable commit SHA.
  • Missing evidence cannot become a false success.
  • Live data is never silently replaced with demo data.
  • The complete suite contains 89 passing tests.
  • The project requires no paid AI service.
  • GitHub credentials remain server-side and are never committed.
  • The dangerous and safe results are independently visible in real GitHub Checks.

What we learned

DataHub metadata and counterfactual execution answer different but complementary questions.

DataHub answers:

What is connected, what does this field represent, and who owns the affected assets?

DuckDB replay answers:

What will actually change if this code is merged?

Metadata alone cannot prove that a behavioral change is dangerous. Execution alone cannot identify every organizational consumer or responsible owner. Combining the two produces evidence that is both technically measurable and organizationally actionable.

We also learned that trustworthy agents need visible provenance. ShadowGraph labels every source, validates immutable identities, bounds expensive operations, and returns an inconclusive result rather than claiming safety when required evidence is missing.

The safe-change scenario was equally important. A safety system that blocks every important change will quickly be bypassed. Demonstrating a real false-positive-free pass made the project significantly stronger.

What's next for ShadowGraph

The next step is expanding ShadowGraph from the reference dbt project to additional real-world repositories and transformation systems.

Planned improvements include:

  • Native webhook-driven analysis across multiple repositories
  • Richer column-level lineage for dashboards, metrics, and feature stores
  • Additional replay adapters for warehouse engines
  • Configurable policies based on asset criticality and ownership
  • Displaying successful before-and-after comparisons for safe changes
  • Approved, idempotent decision writeback into DataHub
  • Historical impact evidence for future developers and agents
  • Notifications for affected owners
  • A hosted demonstration environment
  • Support for additional orchestration and schema formats

Our long-term goal is for ShadowGraph to become a reusable organizational safety layer: every important data change should arrive with evidence showing what it affects, how behavior changes, who owns the consequences, and whether it is safe to merge.

Built With

Share this project:

Updates