ChangeSafe

Evidence-led safety for analytics schema changes before merge.

Comment

Inspiration

A schema change can look local while breaking models, metrics, dashboards, and the teams that trust them. Code review rarely contains ownership, governance, query usage, semantic dependencies, and field-level lineage in one place. We built ChangeSafe to ask the catalog what a change means before generating or publishing anything.

What it does

ChangeSafe is an evidence-led safety agent for analytics schema changes. It starts with an allowlisted DataHub schema, lets a reviewer choose any returned field, retrieves field-scoped context, traces upstream and downstream dependencies, classifies six impact areas, calculates a transparent deterministic risk score, and generates a conservative seven-file dbt/SQL compatibility package.

Twelve blocking checks verify the request, metadata, SQL, YAML, output contract, paths, rollback order, and exact manifest bytes. The workflow always pauses for the accountable owner. In the public competition mode, approval creates a non-mutating receipt and downloadable patch—nothing is silently written.

The official showcase-ecommerce catalog supplies 55 concrete order_details fields. The demo proves three distinct flows:

  • Rename cust_email to primary_email: 6 upstream and 25 downstream relationships.
  • Remove order_status: 6 upstream and 27 downstream relationships.
  • Change order_total to VARCHAR(320): 6 upstream and 31 downstream relationships.

Lineage is deliberately honest. Exact routes name both returned fields. Endpoint-only and dataset-level relationships disclose missing precision instead of inventing a column mapping.

How we built it

  • DataHub Agent Context Kit for schema, lineage, governance, ownership, and usage context.
  • FastAPI, Pydantic, SQLite, UUIDv7 run IDs, and resumable server-sent events.
  • React 19, TypeScript, Vite, semantic HTML, and a responsive evidence-first interface.
  • Deterministic policy and reviewed templates for migration SQL, YAML, tests, rollback guidance, and PR evidence.
  • sqlglot, safe YAML parsing, semantic type validation, path confinement, and SHA-256 manifests.
  • Optional read-only Snowflake aggregate validation with identity and allowlist checks.
  • GitHub/DataHub publication adapters protected by owner approval, immutable intent, and crash-safe checkpoints.
  • pytest, Vitest, Playwright, Ruff, mypy, dbt, Docker, and GitHub Actions.

Challenges we ran into

The hardest problem was preserving truth across live and recorded modes. The public demo replays a checksum-pinned DataHub evidence bundle through the real API, persistence, event stream, policy, generation, verification, approval, and patch-download pipeline. It clearly labels Recorded DataHub evidence, Preview only, and Production rows not queried.

A second challenge was incomplete lineage. DataHub may return an endpoint or asset path without every intermediate column mapping. ChangeSafe shares one precision-aware route model across the graph, drawer, and accessible list so uncertainty is visible instead of hidden.

We also designed publication recovery so a restart cannot duplicate a pull request or mix decision identities after a partial external write.

Accomplishments that we're proud of

  • Schema-driven keyboard selection across exactly 55 official fields.
  • Three operation-specific flows with field-specific evidence and generated bytes.
  • Six evidence-led impact categories and a traceable factor ledger.
  • Seven sealed artifacts and 12/12 blocking checks.
  • Fail-closed handling for unknown fields, rename collisions, stale evidence, invalid types, empty/all-null warehouse results, and lost responses.
  • Responsive desktop/mobile behavior, keyboard focus return, reduced-motion support, and durable refresh recovery.
  • Apache 2.0 source with reproducible Docker setup and a permanent hosted demo.

What we learned

Metadata becomes much more valuable when it is executable decision context rather than a passive catalog. A trustworthy agent needs provenance, bounded authority, deterministic policy, semantic validation, human approval, durable receipts, and honest uncertainty as much as it needs generation.

What's next for ChangeSafe

Add organization authentication and shared job storage, run owner-controlled aggregate validation against a read-only Snowflake environment, and extend the same evidence contract to additional data quality and ML change workflows.

Built With

Share this project:

Updates