BLACKOUT

Chaos engineering for your data stack. BLACKOUT uses DataHub to simulate failures before production, map blast radius, find structural SPOFs, and turn every test into resilience memory.

Comment

⚡ BLACKOUT

Break it in simulation, before it breaks you in production.

BLACKOUT is chaos engineering for your data stack — powered by the DataHub context graph.

Software teams have chaos engineering.

Data teams usually discover hidden dependencies after a table disappears, a pipeline breaks, a warehouse object changes, or dashboards stop receiving fresh data.

BLACKOUT asks a different question:

What would break if this data asset failed right now?

Instead of intentionally disrupting production infrastructure, BLACKOUT uses the live DataHub context graph as a safe simulation substrate, propagates a simulated failure through real lineage, measures downstream impact deterministically, plans multi-experiment chaos campaigns, identifies structural single points of failure, and writes verified resilience knowledge back into DataHub.

🔗 Explore BLACKOUT

🌐 Interactive Public Blackout Map
💻 GitHub Repository
🎥 Product Video
🧩 Upstream DataHub MCP PR #204

BLACKOUT

💥 Inspiration

Modern software engineering has learned an important lesson:

Do not wait for production to teach you where the system is fragile.

Chaos engineering tools deliberately test software systems before real incidents happen.

But in data infrastructure, the equivalent workflow is still uncommon.

A data team may have:

  • hundreds of datasets,
  • transformation jobs,
  • warehouses,
  • dbt models,
  • dashboards,
  • owners,
  • domains,
  • runbooks,
  • and deeply connected lineage...

...yet still not know which dependency will create the largest operational cascade until something actually breaks.

That led to the core idea behind BLACKOUT:

Netflix breaks software on purpose.
BLACKOUT breaks the data stack in simulation.

DataHub already knows how the stack is connected.

BLACKOUT turns that context graph into something new:

a failure simulation environment.

🧠 The 30-second mental model

           LIVE DATAHUB CONTEXT GRAPH
                     │
                     ▼
             SELECT A FAILURE
                     │
                     ▼
          SIMULATE THE CASCADE
                     │
                     ▼
      MEASURE VERIFIED DOWNSTREAM IMPACT
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       CASCADES     SPOFs     PROTECT
          │
          ▼
              BLACKOUT MAP
                     │
                     ▼
             RESILIENCE MEMORY
                     │
                     ▼
                  DATAHUB

DataHub is present at both ends of the loop.

It provides the context BLACKOUT needs to simulate failure — and becomes the place where BLACKOUT stores what it learned.

🏗️ Architecture

BLACKOUT Architecture

The architecture deliberately separates AI interpretation from mathematical truth.

DataHub Context Layer

BLACKOUT reads live DataHub context including:

datasets → jobs → storage → warehouse → models → BI

and organizational metadata such as:

owners · domains · tags · documents

Chaos Campaign Agent

An optional Anthropic-powered planner selects experiments and creates:

  • campaign roles,
  • hypotheses,
  • rationale,
  • expected learning.

Verified Fact Firewall

The agent cannot simply invent quantitative evidence.

BLACKOUT maintains deterministic fact provenance across:

candidate_local · relational · selection · campaign

Claude references verified facts rather than becoming the source of truth.

Deterministic BLACKOUT Engine

Python owns every quantitative and structural conclusion:

  • lineage traversal,
  • downstream casualties,
  • BI casualties,
  • cascade depth,
  • pairwise failure-surface overlap,
  • marginal campaign contribution,
  • directed-graph dominators,
  • structural dependency coverage.

Resilience Memory

After explicit confirmation, verified findings can be written back to DataHub as resilience metadata.

🤖 Claude decides WHY. BLACKOUT determines WHAT breaks.

This became one of the most important architectural principles in the project.

┌─────────────────────────┐
│         CLAUDE          │
│                         │
│ experiment selection    │
│ hypothesis              │
│ rationale               │
│ expected learning       │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│  VERIFIED FACT FIREWALL │
│                         │
│ evidence references     │
│ provenance              │
│ numeric validation      │
│ semantic guardrails     │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│    BLACKOUT ENGINE      │
│                         │
│ casualties              │
│ BI impact               │
│ cascade depth           │
│ overlap                 │
│ marginal contribution   │
│ dominators              │
└─────────────────────────┘

The LLM never computes the cascade.

If Claude says something quantitative, BLACKOUT must be able to trace that claim back to deterministic evidence.

BLACKOUT also guards against unsupported semantic escalation.

For example:

High downstream overlap does not automatically mean structural SPOF.

And missing metadata about monitoring, retries, or failover cannot magically become proof that those controls exist.

🧨 What happens during a chaos simulation?

A user can run:

blackout run --asset "orders"

BLACKOUT resolves the asset against DataHub, obtains its live context and lineage, and safely simulates its disappearance from the dependency graph.

For one validated example using the DataHub showcase graph:

postgres / orders

Downstream casualties: 41
BI casualties:         31
Cascade depth:          9
Affected platforms:     8

This is not a probability.

It is deterministic reachability over the observed lineage graph.

Conceptually:

$$ Blast(a) = \left|Reach^{+}(a)\right| $$

where (Reach^{+}(a)) represents downstream assets reachable from failed asset (a).

🗺️ Blackout Map

Every simulation can produce a self-contained interactive HTML artifact showing how the simulated outage propagates through the stack.

The map makes the cascade visually understandable:

Postgres
   │
   ▼
Spark
   │
   ▼
S3
   │
   ▼
Snowflake
   │
   ▼
dbt
   │
   ├────────► Looker
   ├────────► Power BI
   └────────► Tableau

The public artifact can be explored here:

👉 Explore the public Blackout Map

The public map uses BLACKOUT's sanitized synthetic fixture.

Live BLACKOUT runs operate against the organization's actual DataHub context graph.

🧪 Autonomous Chaos Campaigns

A single failure simulation answers:

What happens if this asset fails?

But a resilience program needs another question:

What should we test first?

Running:

blackout campaign

creates an autonomous campaign of up to three experiments.

Each experiment keeps several concepts explicitly separate:

Verified Impact

The asset's standalone failure impact.

Verified Evidence

Intrinsic and relational facts available before campaign execution.

Campaign Contribution

The new failure surface added after previously selected experiments.

Organizational Context

Owners, domains, tags, and organizational documentation.

BLACKOUT Memory

Knowledge generated by earlier BLACKOUT simulations.

That distinction matters.

An asset may have:

Standalone casualties: 41
Standalone BI impact:   31

while adding only:

Marginal casualties: +5
Marginal BI impact:  +0

after another experiment has already exercised most of its downstream surface.

The campaign therefore avoids confusing large individual blast radius with useful incremental test coverage.

For failure surfaces (A) and (B), BLACKOUT can compare overlap using:

$$ J(A,B)=\frac{|A\cap B|}{|A\cup B|} $$

while marginal contribution is computed against the actual ordered set of experiments already selected.

⚠️ High blast radius ≠ Structural SPOF

High Blast vs Structural SPOF

This became one of the most important discoveries while building BLACKOUT.

A node can produce a huge cascade without being structurally unavoidable.

High blast radius

A failure reaches many downstream assets.

Structural SPOF

Every relevant path to downstream targets depends on that node.

BLACKOUT therefore uses cycle-safe directed-graph dominator analysis rather than simply ranking nodes by casualty count.

In the DataHub showcase graph:

postgres / orders

Downstream casualties: 41
BI casualties:         31

Classification:
HIGH BLAST ONLY
NOT a structural SPOF

But:

Snowflake / ORDER_DETAILS

Blast radius:          34
Dominated BI targets:  14
BI platforms:           3

Classification:
STRUCTURAL SPOF

That distinction prevents BLACKOUT from turning basic lineage impact analysis into an invented "criticality score."

🛡️ Protection Planner

Once structural dependencies are known, BLACKOUT can answer:

Where should resilience work start?

blackout protect --budget 3

The protection planner greedily maximizes structural BI dependency coverage.

In one validated run:

Structural BI targets represented: 14 / 14
Coverage saturated

This is intentionally not described as probabilistic risk reduction.

BLACKOUT does not currently know the probability that an asset will fail.

It knows the graph structure.

So it reports what the evidence actually supports.

🔁 DataHub becomes resilience memory

Resilience Loop

A catalog usually tells you what exists.

BLACKOUT adds another question:

What have we already learned about how it fails?

After explicit user confirmation, BLACKOUT can write resilience knowledge back into DataHub, including:

BLACKOUT critical tag
Last simulated timestamp
Downstream casualties
BI casualties
Cascade depth
Deterministic result hash
Failure Playbook

Then:

blackout status --asset "orders"

can answer whether that asset has previously been chaos-tested.

BLACKOUT deliberately distinguishes:

Organizational Context

from:

BLACKOUT Memory

A Failure Playbook created by BLACKOUT means:

"BLACKOUT has simulated this asset before."

It does not mean:

"The organization historically experienced a production incident here."

Preserving that provenance became an important trust feature.

🧩 Why DataHub is essential

BLACKOUT is not a chatbot sitting next to DataHub.

Without the DataHub graph, there is no BLACKOUT simulation substrate.

BLACKOUT uses DataHub in four different ways:

1. Live dependency graph

Lineage supplies the topology used by the deterministic simulation engine.

2. Organizational context

Owners, domains, tags, and documents provide context around the graph.

3. Agent grounding

The campaign agent receives verified DataHub-derived evidence rather than unrestricted prose.

4. Resilience memory

Confirmed simulation results return to DataHub so future analysis can distinguish organizational metadata from previous BLACKOUT knowledge.

The loop is:

Context in → simulation → verified learning → resilience memory out.

🛠️ How I built it

BLACKOUT is a Python 3.11 package with a real CLI:

blackout doctor
blackout config
blackout run --asset "orders"
blackout campaign
blackout status --asset "orders"
blackout spofs
blackout protect --budget 3

The major internal components include:

src/blackout/
├── agent.py          # Anthropic planning + factual/semantic firewall
├── campaign.py       # Multi-experiment chaos campaigns
├── cascade.py        # Cycle-safe deterministic cascades
├── dominators.py     # Formal structural SPOF analysis
├── protection.py     # Structural protection planning
├── memory.py         # DataHub resilience memory
├── mcp_client.py     # DataHub MCP integration
├── render_map.py     # Self-contained Blackout Map
├── doctor.py         # Product/environment diagnostics
└── cli.py            # Public BLACKOUT interface

The public repository also includes a sanitized synthetic lineage fixture so the test suite is reproducible without publishing private catalog metadata.

The current public test suite contains:

177 passing tests

and the package has been validated through:

Fresh Python 3.11 virtual environment
Normal pip installation
Wheel build
sdist build
Fresh wheel installation
Offline CLI smoke tests
Ruff validation

🧗 Challenges I faced

The real DataHub graph is not a DAG

One of the first assumptions I had to remove was that lineage would behave like a perfect acyclic pipeline.

The real graph contained cycles and even a self-loop.

That meant cascade traversal and structural analysis had to be explicitly cycle-safe rather than relying on DAG assumptions.

Large blast radius is not the same thing as structural criticality

Initial simulations made some upstream tables look extremely important because they produced large cascades.

But dominator analysis showed that several high-blast assets were not structurally unavoidable.

That changed the product architecture.

BLACKOUT now separates:

impact
from
structural dependence

instead of collapsing both into a generic risk score.

Making the AI useful without letting it redefine reality

The first autonomous campaigns exposed a subtle problem:

Claude could see a true number in relational evidence and repeat it in a context where the value had different semantics.

That led to the Verified Fact Registry.

Facts now retain provenance across:

candidate_local
relational
selection
campaign

Later, campaign marginal contribution had to become selection-scoped, because the marginal value of an experiment depends on which experiments came before it.

This was one of the hardest and most valuable engineering lessons in the project:

A number is not enough. Truth also requires context and provenance.

Separating organizational history from BLACKOUT's own memory

BLACKOUT had previously written Failure Playbooks into DataHub.

When Claude later saw those documents, it initially interpreted them as evidence that the organization had historically known the asset was fragile.

That was logically incorrect.

I introduced explicit separation between:

ORGANIZATIONAL CONTEXT

and:

BLACKOUT MEMORY

so knowledge created by BLACKOUT can never silently masquerade as pre-existing organizational history.

📚 What I learned

The biggest lesson was that metadata becomes much more powerful when treated as an executable model of the organization rather than passive documentation.

DataHub already contains enough context to answer operational questions that are rarely asked of a catalog:

What fails together?

Which experiment adds a genuinely new failure surface?

Which node merely has high blast radius?

Which node is structurally unavoidable?

What has already been tested?

I also learned that trustworthy agents need more than prompt engineering.

For BLACKOUT, useful AI required:

  • typed structured outputs,
  • deterministic computation,
  • evidence references,
  • fact provenance,
  • numeric validation,
  • semantic guardrails,
  • explicit memory provenance.

The result is a system where AI can still reason creatively without becoming the authority on facts.

🏆 What I'm proud of

BLACKOUT evolved from a simple failure-propagation idea into a complete resilience loop:

OBSERVE
   ↓
SIMULATE
   ↓
MEASURE
   ↓
PLAN
   ↓
UNDERSTAND STRUCTURE
   ↓
REMEMBER
   ↓
OBSERVE AGAIN

The project now includes:

Live DataHub graph extraction
Cycle-safe cascade simulation
Animated Blackout Maps
Autonomous chaos campaigns
Fact-referenced AI reasoning
Selection-scoped campaign evidence
Structural SPOF analysis
Protection planning
Resilience memory
Public CLI and installable package
177 passing tests
Public interactive artifact
Apache-2.0 open-source repository

🌍 Open-source contribution

While building BLACKOUT, I found a generic gap in the DataHub MCP mutation workflow.

add_tags requires the target tag entity to already exist.

I implemented an idempotent:

ensure_tag

mutation tool in the upstream acryldata/mcp-server-datahub project.

The contribution includes creation, reuse, mutation gating, tests, linting, typing, and live validation.

Public upstream pull request:

acryldata/mcp-server-datahub#204 — feat: add idempotent ensure_tag mutation tool

The PR is currently a public upstream pull request; I am not claiming it has been merged or accepted.

🔒 Safety model

BLACKOUT sounds destructive.

Its implementation is deliberately not.

BLACKOUT does not intentionally inject failures into production infrastructure.

Failures are simulated over DataHub metadata lineage.

The current system does not:

shut down databases
delete tables
stop pipelines
break dashboards
inject production outages

DataHub mutations are limited to resilience metadata and occur only after explicit confirmation.

The agent does not execute graph mathematics.

Python does.

🚧 Current boundaries

BLACKOUT intentionally does not claim capabilities that are not implemented.

Current boundaries include:

  • failure simulation operates on metadata lineage rather than real infrastructure disruption,
  • results depend on DataHub lineage quality and completeness,
  • structural analysis is graph-structural, not probabilistic failure prediction,
  • there is currently no Monte Carlo engine,
  • the protection planner measures structural coverage, not probabilistic risk reduction,
  • Anthropic is optional for campaign planning,
  • live BLACKOUT usage requires DataHub.

These are explicit product boundaries, not hidden assumptions.

🚀 What's next

The long-term opportunity is to turn BLACKOUT into a continuously improving data resilience control plane.

Future directions include richer failure scenarios, historical resilience comparison, additional experiment strategies, stronger operational evidence ingestion, and eventually controlled integrations with real testing environments.

But the core principle should remain unchanged:

AI may decide what deserves investigation.
Deterministic evidence decides what is true.

⚡ BLACKOUT

Break it in simulation, before it breaks you in production.

GitHub:
https://github.com/jpablortiz96/blackout

Interactive Blackout Map:
https://jpablortiz96.github.io/blackout/

Video:
https://youtu.be/lRTRgnHlhPg

License: Apache-2.0

Built solo by Juan Pablo Enriquez Ortiz for Build with DataHub: The Agent Hackathon.

Built With

Share this project:

Updates