silentbreak

• 

• 

• 

• 

• 

• 

Inspiration

Three kinds of data failure, and only one is expensive.

Loud — dbt run errors, a column goes missing. Cheap: caught in minutes.

Quiet — a row count visibly craters. Moderate: someone notices by lunch.

Silent — SUM() skips nulls and revenue understates by 8%. The build is green, every test passes, and the dashboard renders a number that looks completely reasonable. Expensive: found in weeks, if ever.

Every lineage tool answers "what depends on this?" Nothing answers "what would break, and would anyone find out?"

That second question is where the expensive failures live. A dashboard showing a plausible wrong number is worse than one that's obviously broken, because nobody investigates it.

What it does

I applied the most ordinary change imaginable to a 22-model dbt warehouse: a new payment method writes NULL to amount instead of 0. A different team ships it on a Tuesday. dbt run → exit 0 (no error) · dbt test → exit 0 (no test fired) · revenue → -7.93% across every downstream mart

Nothing broke. Nothing alerted. Detection lag: infinite.

silentbreak finds those paths, proves them by execution, and closes them:

1. Score — deterministic blast-radius × fragility over the whole dbt graph, no LLM. It independently ranked the exploit corridor mart_revenue → int_order_revenue → stg_payments as the top 3 from graph structure alone.
2. Red agent — reads compiled SQL, proposes a plausible change, classifies each hop ERROR/DEGRADE/SILENT, and must quote the exact SQL construct responsible. Every citation is verified as a literal substring of the real compiled SQL; ungrounded citations are rejected.
3. Prove — actually mutates the seed data, runs dbt, diffs every mart. Detection lag is measured, not asserted.
4. Blue agent — generates a real dbt test, then re-applies the identical mutation to confirm it fires. ∞ → 1.
5. Write back — tags, descriptions, ownership, and detection-lag properties land on real DataHub entities. 16 datasets tagged silentbreak:silent-risk.
6. Cold verifier — a separate subprocess receiving only the mutation fact, never the red agent's reasoning, re-deriving the lag from scratch.

How we built it

Python for everything touching dbt, DuckDB, and DataHub; React for a static dashboard reading a committed run.json. OpenAI with strict JSON-schema structured outputs.

Every LLM call is SHA-keyed and cached to disk, and the cache is committed — the run replays with --offline and no API key. That's the reproducibility mechanism, not an optimization: it pins the exact reasoning behind every published number.

manifest.py reads dbt's compiled SQL directly rather than depending on DataHub for it. That's the source of truth for scoring and citation-grounding; DataHub supplies the catalog view and receives the findings.

Challenges we ran into

Anti-tautology. The naive design scores fragility, hardens, re-scores, and reports the number dropped — which proves nothing, since the hardening step assigns the very owners that fragility penalizes. So the headline metric is detection lag, measured by execution, and the verifier runs in a genuinely isolated subprocess that never sees the red agent's reasoning.

A silent failure inside the silent-failure detector. Against a live DataHub I found my own URN construction was wrong — writes reported success while creating orphan entities instead of updating real ones. Green, confident, and landing nowhere anyone would look. Invisible without a live instance. The irony was not lost on me.

dbt test results were never actually being read. I was scraping a console format dbt doesn't emit, so dbt_test_failures came back empty every time — meaning hardening_effective was always false and the ∞ → 1 result never appeared. Fixed by parsing run_results.json instead of console text.

Accomplishments that we're proud of

The load-bearing claim is real and measured end to end: a mutation that runs green and silently moves revenue, then a generated test that makes the same mutation fail loudly.

Also: score.py re-derived the exploit corridor from graph structure alone, without being told where the flaws were. And 170 metadata events ingested with the findings written back onto real catalog entities.

What we learned

Verification is worth more than accuracy. My red agent scored 10/10 — but every candidate both predicted and measured INFINITE, so a predictor that always guesses "silent" scores identically. The number is real; the discriminative power isn't demonstrated. Reporting that honestly is worth more than the 10/10.

The cold verifier disagreed with my own measurement 4 times out of 10. I'm reporting that rather than tuning it until it agrees — a verifier that always agrees is decoration.

Two fragility terms are hardcoded zero (distinct query authors, recent schema change) because the inputs genuinely aren't available. Labelled as zero, not faked with proxies.

What's next for silentbreak

Sampled mutations — affect 5% of rows rather than 100% — so severity separates truly SILENT from merely DEGRADE. Some current mutations move revenue so far that nobody would miss them, which is a different (cheaper) failure class than the one this targets.

Then running against DataHub's stock sample data. Findings on data I didn't author is the strongest possible answer to "you seeded the mess, then found the mess."

Built With

• docker
• graphql
• openai
• pytest
• python
• react
• sql
• typer
• typescript
• vite