SchemaGuard

See tomorrow's breaking changes before today's merge.

Comment

Inspiration

Friday afternoon. A backend engineer renames one field. CI is green. Unit tests pass. The PR is approved. Three hours later the mobile app crashes — it still depended on the old contract.

API changes fail silently because consumers live outside the author's PR. Diff tools answer what changed. Chatbots answer with an opinion. Neither answers the question a Staff Engineer actually asks in review: what is the safest way to ship this?

SchemaGuard is that review — before merge.

What it does

Give SchemaGuard a base and proposed OpenAPI 3.x document (or open the zero-setup rename fixture), and it:

  1. validates intake (OpenAPI 3.x, size limit, disclaimer),
  2. runs deterministic break rules over component schemas (rename, remove, type narrow, required-add, enum-remove),
  3. builds an Evidence Package with findings, known consumers (fixtures), and a Compatibility Risk Estimate,
  4. prepares Decision Alternatives and a Decision Confidence Matrix (hard cut vs compatibility alias/adapter),
  5. supports What-if so the engineer can apply the safer strategy and see risk / SemVer / rollout update,
  6. emits a Codex handoff prompt grounded only in that evidence.

A real demo, nothing to paste. PR #482: Payment.amount → Payment.total. SchemaGuard reports Changes requested, Compatibility Risk 92 · Critical, three cited consumers (iOS Checkout, Android Checkout, Ledger Export). Hard rename means Major and high release risk. Compatibility alias means Minor and low risk. Apply What-if — estimate drops to 8 · Low, SemVer 2.5.0 · Minor, rollout Canary. Copy the Codex prompt and implement the alias.

One product story, three failure classes. Fixtures span rename breakage, type/enum high-risk breaks, and a safe additive change that must not page as a break.

How we built it

Deterministic where it counts, model where it helps. Rules and risk math never depend on a model. The Evidence Package is the only payload an LLM may see. buildArchitectPrompt and validateArchitectOutput enforce allowlists on finding IDs and recommendation types — a bad model output is rejected, not trusted.

Codex accelerated the workflow. Multi-file evidence engine (lib/analyze.ts), architect trust boundary (lib/architect.ts), fixture demos, SSE milestone stream, and the PR-shaped Next.js UI were built with Codex using GPT-5.6. Key product decisions — Trust View early, What-if as a core beat, Compatibility Risk Estimate as a relative aid — stayed locked against scope creep.

Trust boundary (architecture):

OpenAPI → Diff → Rules → Evidence Package → Prompt Builder → GPT-5.6 → Output Validator → Decision Artifacts

GPT cannot invent breaks (forbidden_reasoning). The offline path still produces a complete Staff-Engineer-shaped review without calling a model.

Challenges we ran into

  • Rename detection must distinguish remove + add from a true rename — matching type, format, and description keeps the headline demo honest.
  • Consumer coverage for pasted specs must stay unknown so the product never pretends to know clients it has not seen.
  • What-if had to feel like a release decision, not a schema editor — risk, SemVer, and rollout move together when the safer strategy is applied.
  • Streaming milestones must reflect the actual analysis (finding counts, estimate), not decorative progress text.

Accomplishments that we're proud of

  • A coherent PR review UX: problem → evidence + trust → alternatives + matrix → What-if → Codex → confident approve.
  • Rules decide whether; architect decides what to do — encoded in types and validators, not just copy.
  • Three live fixtures covering distinct break classes, each rebuildable on every serverless instance.
  • Zero-setup judge path with no secrets and no model key.

What we learned

  • Constraining the model makes it more trustworthy: one Evidence Package with refusals beats a free-form chat over raw OpenAPI.
  • The valuable signal is the decision, not the diff. Pairing findings with alternatives and What-if turns “this renamed” into “ship the alias.”
  • A deterministic evidence floor is what lets GPT-5.6 safely do more.

What's next for SchemaGuard

  • Local $ref resolution and operations-level path diffs.
  • TTL-backed store (e.g. Redis) for multi-instance analyses.
  • Live consumer telemetry hooks; keep unknown coverage honest until wired.
  • Optional server-side GPT-5.6 call on the architect path with validator-gated fallback.

Built With

Share this project:

Updates