Schema Dam

Keep unaffected data products running when schemas break.

Comment

When a schema breaks, do not stop everything. Contain only the real blast radius.

Inspiration

Schema registries are good at answering one question: Is this new schema compatible with the previous one?

But during a real incident, operators need a different answer: Which downstream products actually use the changed field, and what should happen to each of them right now?

Without that context, teams are often forced into a bad choice. They either allow potentially corrupted data to continue flowing or stop an entire pipeline, including products that are not affected.

I built Schema Dam to create a third option: use DataHub’s schema, column-level lineage, identity, and ownership context to contain only the affected paths while everything else keeps running.

What it does

Schema Dam is a bounded data-operations agent and control plane built around a simple loop:

READ from DataHub → ACT on the runtime → WRITE the verified result back to DataHub

  1. Read connected context. The agent reads schema fields, column lineage, asset identity, and ownership through the official DataHub MCP Server. It combines that context with reviewed runtime bindings to determine which downstream products actually depend on the changed field.

  2. Act on the real blast radius. The agent invokes deterministic controller tools to pause only affected input boundaries, activate configured fallbacks, protect companion inputs of multi-input consumers, and validate a repair in shadow. Unaffected products continue processing.

  3. Write the verified result back. After exact-artifact human approval and successful recovery, Schema Dam writes the affected assets, owners, approval evidence, replay result, and final resolution back to DataHub. When supported, it updates the same native DataHub DATA_SCHEMA incident throughout the lifecycle. In environments without that capability, it uses an explicitly labeled tag-and-summary compatibility mode. The next operator or agent inherits the completed decision instead of reconstructing the incident from logs and messages.

In the reference ecommerce incident, a producer changes:

category_code → category.code

Schema Dam contains three affected products while five unaffected consumers continue processing the same log.

3 affected contained · 5 unaffected kept running · 10 consumer-events replayed · 0 production checkpoint movement before approval

How I built it

Schema Dam separates reasoning from runtime authority.

  • The official DataHub MCP Server supplies live schema, column lineage, asset identity, and ownership.
  • A repository Skill constrains the agent’s workflow, evidence requirements, and approval boundary.
  • The agent explains the impact, prepares the repair, and invokes bounded tools.
  • A deterministic Controller MCP owns gates, checkpoints, artifact promotion, replay, reconciliation, and resume.
  • A human must approve the full canary-tested SHA-256 and replay budget before release.
  • A native DataHub DATA_SCHEMA incident, or an explicitly labeled compatibility fallback when native incident APIs are unavailable, carries the verified operational handoff back into the context graph.

The reproducible runtime uses Python, SQLite, an append-only reference log, independent consumer gates, and aggregate plus per-input checkpoints. Each affected consumer accumulates backlog without losing accepted events, while unaffected consumers continue advancing.

For multi-input products, Schema Dam protects every companion input together. This prevents a consumer from combining a repaired stream with incompatible state that continued arriving through another input.

Challenges I faced

Treating missing context as uncertainty

An empty lineage response does not prove that nothing is affected. It may indicate truncation, stale metadata, a failed query, or an unbound downstream asset.

Schema Dam therefore fails closed when lineage is empty, stale, truncated, incomplete, or cannot be mapped to a reviewed runtime boundary.

Protecting multi-input consumers

Pausing only the stream containing the changed field is not always safe. A consumer may continue receiving companion data and build inconsistent internal state.

I had to model containment, checkpoints, canary validation, and replay per consumer and per input while preserving global event order.

Keeping agent reasoning away from irreversible state

Letting an agent directly edit checkpoints or infer approval would make the system difficult to trust.

The agent can explain and request actions, but deterministic code owns every runtime transition. Shadow validation cannot move production checkpoints, and release requires approval matching the exact tested artifact digest.

Recovering from partial external actions

A successful mutation response does not necessarily prove that a pipeline has reached the intended state.

Schema Dam stores desired and observed state separately, verifies external state independently, and reconciles timeouts, retries, controller restarts, and partially applied actions.

Accomplishments

  • Contained three affected product boundaries while five unaffected consumers continued processing.
  • Preserved every accepted event and replayed the affected backlog in order.
  • Kept production checkpoints unchanged throughout shadow validation.
  • Required explicit approval of the full tested artifact SHA-256 before promotion.
  • Protected companion inputs for multi-input consumers.
  • Implemented a durable DataHub incident/write-back lifecycle covering containment, approval waiting, replay, resolution, and reopen.
  • Built a one-command deterministic HTTP-boundary proof and a separate live DataHub-driven agent workflow.

The ecommerce data and SQLite/Python runtime are synthetic reference evidence. They are not presented as production-scale benchmark results.

What I learned

The most important lesson was that schema compatibility and downstream impact are different problems.

A change may be technically compatible but still break a product that relies on the changed field’s meaning, shape, or requiredness. DataHub’s context graph becomes operationally valuable when lineage, ownership, runtime bindings, serving policy, and sink behavior are evaluated together.

I also learned that useful agents do not need unlimited authority. An agent can perform meaningful work by gathering context, explaining decisions, preparing artifacts, and coordinating tools while deterministic systems retain control of irreversible state.

Finally, writing the result back matters as much as reading the initial context. A resolved incident should leave behind enough structured evidence for the next person or agent to understand what changed, what was protected, who approved the repair, and how recovery was verified.

What's next

The next step is to validate the same control contract in dedicated infrastructure:

  • Publish reviewed 100k/1M backlog and multi-hour soak results.
  • Run a target-environment resilience game day covering controller failure, stale lineage, broker rebalancing, partial control failure, restore, and rollback.
  • Pilot Schema Dam with a real data platform team and measure containment time, operator effort, false-positive impact, and recovery confidence.
  • Expand deployable remediation artifacts beyond the reference Python transformation while keeping the same approval and provenance guarantees.

Schema Dam is not trying to replace stream processors, orchestrators, or schema registries. It gives agents a safe way to use DataHub context to coordinate them when a data change escapes into production.

Built With

Share this project:

Updates