Schema Sentinel

Inspiration

A schema migration can compile successfully and still break the meaning of a business metric. For example, replacing customer_id with shopper_id looks like a simple column change. But if multiple shopper identities map to the same CRM customer, a direct substitution changes customer cardinality, attribution, and lifetime-value calculations—even while total revenue remains unchanged. DataHub already knows the surrounding context: schemas, ownership, glossary meaning, documentation, and downstream lineage. I built Schema Sentinel to turn that context into a governed safety decision before a migration is merged.

What it does

Schema Sentinel is a self-hosted schema-migration agent that: Retrieves schema fields, fine-grained lineage, ownership, glossary terms, runbooks, and precedents through the official DataHub MCP server. Evaluates that evidence using a deterministic, versioned policy pack. Fails closed with REVIEW_REQUIRED whenever required governance or lineage evidence is missing. Runs a counterfactual simulation against a synthetic DuckDB/dbt environment. Generates a compatibility model, dbt regression tests, impact analysis, migration instructions, rollback guidance, and a draft PR. Requires explicit approval of the exact safety-case hash before publishing. Records an immutable safety case and creates a post-merge verification receipt. In the main identity-break scenario, unsafe substitution produces: CRM identity coverage of 0 / 3 Three apparent customers instead of two Average customer LTV of $64.83 instead of $97.25 Unchanged total revenue of $194.49 Schema Sentinel generates a compatibility bridge using customer_identity_map, restoring all tested invariants.

How we built it

The backend uses Python, FastAPI, and Pydantic. The review interface uses React and TypeScript. DataHub OSS supplies the context graph, while the official MCP server exposes schema, lineage, entity, and document retrieval. DuckDB and dbt-duckdb provide a reproducible execution environment for counterfactual testing. GitHub publication uses a controlled repository allowlist and preserves strict ordering: the generated branch and commit must succeed before DataHub write-back occurs. Every safety case contains normalized evidence citations, retrieval provenance, policy findings, generated-file hashes, approval state, publication proof, and verification status. Canonical decisions and hashes are deterministic. No model training or API key is required; optional AI-generated prose cannot alter policy, artifacts, simulation results, or hashes. The default fixture mode lets anyone evaluate the complete application without a DataHub account. The same evidence path was also validated against DataHub OSS through the official stdio MCP server in GitHub Actions.

Challenges we ran into

The hardest challenge was distinguishing metadata persistence from metadata readiness. DataHub could accept the seeded aspects while search, ownership relationships, document discovery, or fine-grained lineage were still being indexed. Early integration checks repeatedly started the MCP server while waiting for those indices, producing large logs and inconsistent failures. I fixed this by: Creating referenced glossary and owner entities before their associations. Verifying stored aspects directly at the GMS boundary. Waiting specifically for searchable field lineage. Treating missing MCP evidence as a real fail-closed condition. Starting the expensive stdio MCP analysis only after readiness checks passed. Adding recorded-response and live-contract regression tests. Canonical safety-case hashing also required careful timestamp normalization so that the committed case and post-merge receipt could be independently verified.

Accomplishments that we're proud of

We built a complete, auditable safety loop—not just a metadata demo. Schema Sentinel retrieves real DataHub context through the official MCP server, proves the business impact of an unsafe migration, generates working dbt code and tests, opens a controlled draft PR, preserves an immutable safety case, and verifies the migration after merge. Most importantly, it fails closed when evidence is incomplete and never lets optional AI prose influence the canonical decision.

What we learned

Metadata retrieval alone is not enough for a trustworthy agent. A production-oriented agent should explain which evidence it used, distinguish missing evidence from low risk, prove its recommendation against executable invariants, require human approval, and leave behind an auditable record. The most valuable role for an AI agent in this workflow is not confidently generating SQL. It is connecting organizational context to deterministic safeguards and making unsafe shortcuts difficult to approve accidentally.

What's next for Schema Sentinel

Next, we would expand Schema Sentinel from the bounded Northstar scenarios into reusable policy packs for broader schema changes, warehouses, and transformation frameworks. We also plan to integrate it directly into CI and data-contract workflows, add organization-specific approval policies, and contribute its governed schema-change methodology as a reusable DataHub Skill—turning safety cases into a standard part of everyday data engineering.

Built With

• datahub
• dbt
• docker
• duckdb
• fastapi
• mcpserver
• python
• react
• typescript