Lineage Fuzzer

Break data safely before bad data breaks production.

Comment

Inspiration

Data-quality controls are usually written after an incident. A pipeline can stay green while an amount changes scale by 100×, a partition becomes 45 days stale, or a join key quietly fills with nulls. Types still look valid, jobs still finish, and the failure reaches a dashboard before the team learns that its existing tests did not cover it.

DataHub already knows which assets are connected, who owns them, what their schemas contain, and which assertions exist. I built Lineage Fuzzer to turn that context into a proactive reliability exercise: break an isolated copy on purpose, measure the controls, generate the missing tests, and prove restoration.

What it does

Lineage Fuzzer runs one deterministic, approval-bound semantic fault campaign:

  1. captures an exact six-dataset commerce graph from live open-source DataHub;
  2. validates complete schemas, five lineage edges, ownership, domain, project and sandbox tags, the sandbox=true marker, and three active custom assertions;
  3. predicts the downstream blast radius of faults against fuzzer.raw.orders;
  4. snapshots a disposable DuckDB fixture and injects three different failures: a 100× amount-scale change, a 45-day stale partition, and a 10% customer-key null surge;
  5. compares predicted impact with observed table-checksum changes and measures the controls captured from DataHub;
  6. records the honest baseline—one of three faults detected;
  7. deterministically emits two validated, runnable, read-only SQL controls from the captured schema, measured gaps, and a clean-data profile;
  8. reruns the identical seeded campaign and improves coverage from 33.3% to 100%; and
  9. verifies that all six fixture checksums returned to baseline.

The result proves coverage for these explicit semantic failures on one isolated fixture. It does not claim that a finite campaign proves an entire production stack reliable.

How I use DataHub

DataHub is the campaign's source of truth. The DataHub MCP Server supplies exact entities, schema fields, and direct lineage. DataHub GraphQL reads the custom assertions that define baseline coverage, and supported OpenAPI operations manage the exact allowlisted sandbox catalog.

The live snapshot is bound to verified catalog state, advertised MCP tool schemas, raw-response digests, and current fixture checksums. Stale or incomplete context makes readiness fail and disables execution. A separate reversible proof created and activated a custom assertion, reported a fixed result, reread it, and restored it by verified soft deletion; that proof is kept distinct from the disposable DuckDB campaign.

How I built it

The application uses Python, FastAPI, Pydantic, DuckDB, httpx, the DataHub SDK, and the MCP Python SDK. Its small same-origin HTML, CSS, and JavaScript console shows the captured topology, fault manifest, baseline and improved matrices, generated SQL artifact, and restoration result.

The planner, fault adapters, scoring, and SQL control builder are deterministic. Generated SQL must parse as one read-only statement, reference only the approved fixture table, pass against clean data, and detect its intended fault before it counts as improved coverage.

Challenges I faced

Real MCP responses exposed harmless field-order differences alongside genuinely ambiguous envelopes. I made parsing tolerant of non-semantic order while continuing to reject duplicates, foreign URNs, pagination, missing fields, contradictory metadata, and unknown response shapes.

An approval-bound reset also completed its writes while an older verifier tried to traverse lineage through tombstoned datasets. I replaced that check with exact status reads and proved an idempotent zero-write recovery.

The most important safety challenge was making a destructive-looking demo genuinely bounded. The target is constrained by DataHub metadata, physical path allowlists, an exact manifest approval, single-flight execution, unconditional restoration, and final checksum verification.

What I learned

Metadata is most powerful when it becomes executable policy. The same lineage that predicts business impact can constrain a fault target, and the same assertions that describe current quality can become a measurable coverage baseline. I also learned that “restore” needs a snapshot, an unconditional recovery path, positive postconditions, and evidence—not just a cleanup call.

What I am proud of

  • Three genuinely different seeded semantic faults exercise one identical campaign.
  • Every predicted blast radius matches the observed effect.
  • Measured coverage improves from one of three to three of three with executable SQL.
  • Every fault restores in a finally path and all six final checksums match baseline.
  • The public app is gated by current live DataHub context and exposes no production target.

Limitations and what's next

Faults touch only an isolated DuckDB fixture; the product never corrupts DataHub or a production warehouse. The campaign covers three designed failures rather than every possible data defect. Next I would add duplicate-key, schema-drift, and referential-integrity adapters; use column-level lineage and criticality to rank campaigns; and emit reviewable dbt tests alongside standalone SQL.

Built With

Share this project:

Updates