ProofGate

💡 Inspiration

Every code-review tool we've used is great at suspecting things and terrible at proving them. You open a PR and get a wall of "potential issues" and "possible regressions" — and then you do the real work: check out the branch, write a repro, run it, and discover half of them aren't real.

We wanted to flip that. What if a finding wasn't allowed to reach you until the tool had already reproduced it, fixed it, and independently verified the fix? A finding should mean proof, not a hunch. That idea — proof-carrying pull requests — became proofgate.

🔍 What it does

proofgate is an evidence-first release gate for pull requests. For every alleged regression it runs the exact same reproduction against three isolated git worktrees:

• Base — behavior before the PR
• PR — what the contributor changed
• Patched — behavior after an agent's fix

…and classifies the result causally: confirmed regression, pre-existing defect (so the contributor isn't blamed), resolved, or rejected false positive. A timeout or broken environment never counts as evidence.

Specialized GPT-5.6 agents — a Change Mapper, correctness/security/browser investigators, a Patch Agent, and an independent verifier that trusts no one — do the reasoning. Findings without a reproduction are rejected, not reported. Nothing ever touches the contributor's branch.

🛠️ How we built it

A TypeScript pnpm monorepo:

• contracts — Zod schemas that are the single source of truth for every agent result.
• orchestrator — the deterministic control plane: git worktree isolation, Vitest & Playwright reproduction runners, the causal verdict engine, a better-sqlite3 evidence store, and a Fastify + SSE server.
• dashboard — a React/Vite tribunal that streams the live run.

GPT-5.6 runs every agent through the OpenAI Agents SDK with schema-validated structured output. The golden rule: the model reasons; deterministic code decides and executes. A seeded demo repo + PR ships a real regression, a pre-existing defect, and a deliberate false positive — so the entire tribunal is provable live, with no mocks anywhere in the demo path.

🧗 Challenges we ran into

• Keeping the model on a leash. Agents may only return validated reproduction specs — never shell commands or executable code. Everything runs shell: false, allowlisted, path-contained, and timed out.
• Making a live LLM pipeline reliable enough to demo. A single malformed structured field from one agent could abort an entire run. We added deterministic field normalization (for values we already know) plus a self-repair retry that feeds the validation error back to the model — so no one imperfect response kills the demo.
• Real-world sharp edges: Windows' 260-character path limit vs. nested worktrees, and browser-runner timing that had to separate server boot from the reproduction's own timeout budget.

🏆 Accomplishments that we're proud of

• A full live run classifies all five finding types correctly — including catching a pre-existing defect and rejecting a plausible-looking false positive.
• The verifier is genuinely independent — it re-runs every reproduction itself and never takes an investigator's word for it.
• It's honest by construction: every status on the dashboard links back to a real execution, and the tool never modifies or pushes a branch.

📚 What we learned

Separating intelligence from control is the whole game. Let the model reason freely inside tight, deterministic guardrails and you get something rare in AI tooling: results that are repeatable and trustworthy, not vibes.

🤖 How we used Codex and GPT-5.6

• Codex was our primary engineering collaborator — from concept to a milestone architecture, it implemented the contracts, the deterministic orchestration, git worktree isolation, the reproduction runners, the SQLite evidence store, the dashboard, and the tests, and helped debug and verify along the way.
• GPT-5.6 is the reasoning inside the product — the Change Mapper, the correctness/security/browser investigators, the Patch Agent, and the independent verifiers, all through the OpenAI Agents SDK with typed structured outputs.

🚀 What's next for proofgate

GitHub Checks integration, more scanner rules and languages, a hosted judge-friendly instance, and exportable release reports — turning proof-carrying pull requests into a drop-in gate for any repository.

Built With

• better-sqlite3
• codex
• fastify
• git
• gpt-5.6
• node.js
• openai-agents-sdk
• playwright
• pnpm
• react
• server-sent-events
• sqlite
• typescript
• vite
• vitest
• zod