SchemaProof

SchemaProof is a Codex plugin that tests database migrations across real app/schema compatibility matrices, producing deterministic evidence before deployment.

Comment

SchemaProof — Devpost submission

About the project

Inspiration

AI coding agents can generate a plausible database migration in seconds, but deployment safety is a temporal problem that a normal unit test rarely models.

During a rolling deployment, the old and new application versions may coexist while the database schema changes underneath them. A migration can therefore pass CI, work perfectly with the new application, and still break the old application that remains in production for several minutes. Rollback can create the opposite compatibility requirement.

Existing migration tools are strong at analyzing SQL operations, but I wanted evidence about the real application contract: can the application versions that will coexist during deployment actually run against the schema versions they may encounter?

That question became SchemaProof.

What it does

SchemaProof is an installable Codex plugin and deterministic command-line runner for proving database migration compatibility before deployment.

Given a Git base ref, a head ref, and a deployment policy, it creates isolated temporary worktrees and executes real repository-defined setup and contract commands across an application/schema compatibility matrix:

Application version Old schema New schema
Old application Baseline Critical for migration-first deployments
New application Critical for app-first deployments Target state

SchemaProof supports three explicit policies:

  • migration-first: requires old/old, old/new, and new/new;
  • app-first: requires old/old, new/old, and new/new;
  • rollback-safe: requires all four combinations.

The verdict is never “universally safe.” It is safe or unsafe for the selected deployment policy, and every result is backed by process exits, captured output, and reproducible JSON and HTML reports.

The included zero-dependency demo starts with an unsafe migration that renames users.name to full_name. The new application works, but the old application fails against the new schema with no such column: name. SchemaProof correctly reports UNSAFE FOR MIGRATION-FIRST.

Codex, running GPT-5.6, then uses that evidence to design an expand-and-contract repair: preserve name, add full_name, backfill it, and allow both application versions to coexist. SchemaProof reruns the exact same policy and produces SAFE FOR MIGRATION-FIRST.

The model explains and repairs. The deterministic runner remains the judge.

How I built it

SchemaProof is implemented as a repository-local Codex plugin with a $schema-proof skill and a Node.js ESM CLI.

The runner:

  1. validates the requested Git refs and policy;
  2. creates isolated temporary Git worktrees for the old and new revisions;
  3. reads .schemaproof.json from both refs;
  4. creates fresh temporary SQLite databases for each required matrix cell;
  5. runs repository-defined schema setup and application contract commands;
  6. records exit codes, stdout, stderr, timing, and requirement status;
  7. evaluates only the cells required by the selected policy;
  8. emits terminal, JSON, and reviewable HTML evidence.

The CLI has no runtime package dependencies. It uses Node.js and Git, and the MVP is tested through GitHub Actions on Windows, macOS, and Ubuntu. The demo does not require an account, API key, external database, or production credentials.

Security and scope boundaries were deliberate. SchemaProof exposes only a minimal process environment; additional variables must be explicitly allowlisted. It operates on temporary databases, never connects to production, and runs repository code only from refs the user chooses.

How I used Codex and GPT-5.6

Codex was central to the entire development process, not added after the product was complete.

I used Codex with GPT-5.6 to:

  • research developer-tool pain points and compare several product directions;
  • identify temporal application/schema compatibility as the narrowest high-value opportunity;
  • design the policy model and compatibility matrix;
  • scaffold the Codex plugin, manifest, skill, CLI, fixtures, and tests;
  • implement and repeatedly exercise the isolated Git-worktree runner;
  • inspect the failing matrix cell and design the expand-and-contract repair;
  • validate cross-platform behavior and tighten environment isolation;
  • create the documentation, product identity, release, and demo video.

The key design choice was to separate probabilistic reasoning from the final verdict. GPT-5.6 understands the repository, explains failures, and proposes repairs. Real processes and explicit policy requirements decide pass or fail.

Challenges I ran into

The hardest challenge was making a temporal deployment problem reproducible locally. Each matrix cell needs the correct combination of application revision and schema state without leaking files, databases, or environment variables between runs. Temporary Git worktrees and fresh database paths provided the isolation required for trustworthy evidence.

The second challenge was avoiding a misleading definition of “safe.” A migration can be safe when the schema deploys first but unsafe when the application deploys first, or insufficient for rollback. Modeling named deployment policies made the result precise and actionable.

Cross-platform execution was another constraint. Shell-specific orchestration would make the demo fragile for judges, so the runner uses Node.js process APIs and Git directly. Automated tests now cover argument parsing, policy requirements, environment allowlisting, and the complete unsafe-to-safe integration path.

Finally, the Codex integration had to preserve trust. It would have been easy to let the model infer that a migration “looks safe.” SchemaProof instead gives Codex deterministic evidence and prevents model confidence from overriding a failing required cell.

Accomplishments that I am proud of

  • A judge can clone the repository and run the entire unsafe and repaired demo with npm run demo and no dependency installation.
  • The failure is attributed to the exact required matrix cell instead of a generic migration warning.
  • The same policy produces a red verdict before the repair and a green verdict afterward.
  • Reports are available in terminal, JSON, and HTML formats.
  • The installable Codex plugin and standalone CLI use the same deterministic engine.
  • The MVP has automated tests and cross-platform CI.

What I learned

The most important lesson is that migration safety is not only a property of SQL. It is a property of application behavior, schema state, and deployment order over time.

I also learned that AI developer tools become more trustworthy when they clearly divide responsibilities: use the model for contextual understanding and repair design, but use deterministic execution for claims that must be reproducible.

What's next for SchemaProof

The next steps are PostgreSQL and MySQL adapters, ORM-aware configuration helpers, richer contract-test discovery, pull-request annotations, and first-class CI integration. I also want to extend evidence collection for dual reads/writes and multi-step contract migrations while keeping the same rule: no safety claim without a required, reproducible check.

Try it

Requirements: Git and Node.js 22.13 or newer.

git clone https://github.com/terrizoaguimor/schema-proof.git
cd schema-proof
npm test
npm run demo

The demo writes the unsafe and repaired reports to artifacts/demo.

Built with

Codex, GPT-5.6, OpenAI, Codex Plugins, Codex Skills, JavaScript, Node.js, SQLite, Git, GitHub, GitHub Actions, HTML, JSON, CLI, Database Migrations, DevOps, CI/CD, Software Testing

Try it out links

Built With

  • ai
  • ci/cd
  • cli
  • codex
  • codex-plugins
  • codex-skills
  • database-migrations
  • developer-tools
  • devops
  • git
  • github-actions
  • gpt-5.6
  • javascript
  • node.js
  • openai
  • software-testing
  • sqlite
Share this project:

Updates