Inspiration
AI coding agents are getting very good at producing patches. The part that kept bothering me was what happens next.
An agent can write a convincing diff and then explain, in equally convincing language, why its own change works. But that explanation comes from the same system that authored the patch. It is useful context, not independent evidence. I wanted a developer tool that treated an AI-generated repair as an untrusted candidate until the repository itself proved it.
That became the question behind PatchProof:
What if the patch had to prove itself?
I deliberately rejected the easier version of this project: a generic autonomous bug fixer that runs a few commands and reports success. Codex can already inspect repositories and make changes. PatchProof needed a different reason to exist. Its job is to put a reliability layer around agent-generated repairs by separating model work from command-backed proof.
What it does
PatchProof is a local developer tool that turns one actionable bug report into a repair with reproducible evidence.
A run begins with a trusted local Git repository and a concrete bug description. PatchProof inspects the repository, determines whether it has a supported verification capability, and creates an isolated Git worktree. The user's original checkout is not modified during diagnosis, test generation, patching, or verification.
Before Codex changes production code, PatchProof runs the existing baseline. A repository with a failing baseline is blocked instead of being “repaired” on top of uncertain state.
PatchProof then starts a Codex SDK thread pinned to GPT-5.6 Sol. Codex diagnoses the defect and writes a focused regression test. The important part is what happens next: PatchProof preserves that generated test while restoring the original production code, then independently executes the test. The regression must fail before the production patch exists.
Only after that failure has been captured does Codex produce a minimal production repair. PatchProof replays the same target test, runs the configured full verification suite, and records every authoritative command result.
A run is VERIFIED only when all four conditions hold:
- the original baseline passed;
- the new regression failed before the production patch;
- the same regression passed after the patch; and
- the full verification suite passed.
If Codex produced a change but any proof condition is missing, the result is UNVERIFIED. If PatchProof cannot safely begin or continue—because the repository is invalid, Docker is unavailable, dependencies cannot be provisioned, or the baseline fails—the result is BLOCKED. The model never promotes its own work to VERIFIED.
Each terminal run produces an evidence package containing the readable report, structured manifest, commands, arguments, exit codes, durations, stdout/stderr logs, diff, apply-ready patch, file classifications, risk notes, and checksums. The UI exposes these artifacts directly instead of replacing them with an agent summary.
After verification, applying the patch is still a separate guarded action. PatchProof checks that the source HEAD and working tree still match the inspected state and requires explicit confirmation. For a verified repository with an authenticated GitHub remote, it can alternatively create an isolated delivery branch and open a pull request—again only after explicit approval.
Supported workflows
I kept support intentionally bounded rather than claiming that GPT-5.6 makes every repository automatically supported.
PatchProof currently provides built-in verification profiles for:
- Node.js and TypeScript projects using npm, pnpm, or Yarn;
- standard Python projects using pip and pytest;
- a single Git-root Go module using the official Go container workflow; and
- structured browser bugs in Node applications using Playwright.
The browser workflow accepts typed reproduction actions and assertions, then replays the same scenario before and after the patch. It captures screenshots, traces, assertions, console output, and network observations. Visual similarity alone is not treated as proof.
PatchProof also supports committed custom verification profiles, but they are deliberately restricted. A custom profile is normalized and SHA-256 hashed. The user must confirm the repository name and exact committed profile hash. Commands are locked to allowlisted Node, Python, and Go runtime policies. Profiles cannot choose arbitrary Docker images, shell interpreters, host paths, mounts, secrets, lifecycle hooks, or weaker network/write policies.
The project does not claim universal language support, arbitrary monorepos, private Go modules, Poetry, Conda, CGO, or browser verification outside the accepted Node capability. A new ecosystem becomes supported only after its provisioning, test discovery, isolation, mutation rules, cleanup, and failure behavior are independently validated.
How I built it
PatchProof is a TypeScript workspace built around four main boundaries:
- a React and Vite local interface;
- a Fastify orchestration API;
- shared Zod runtime contracts; and
- a Core package containing repository inspection, profiles, the state machine, process execution, worktree isolation, Codex integration, Docker execution, evidence generation, and delivery guards.
The state machine separates lifecycle from proof outcome. Completing a run does not imply that the repair was verified. Stage transitions, cancellation, timeouts, cleanup, proof updates, and terminal classifications are explicit.
External commands never use interpolated shell strings. Executables and arguments are passed separately, and each process result records timestamps, duration, timeout state, cancellation state, exit code, stdout, and stderr. Repository contents and bug descriptions are treated as untrusted input.
Git worktrees provide source isolation. Docker provides bounded execution profiles. Run artifacts live under a run-owned local directory instead of a database. Server-Sent Events stream stages and evidence to the dashboard while the filesystem remains the durable source of truth.
The public Vercel site is intentionally not the verifier. The real product requires trusted access to local repositories, Docker, Git, the user's Codex authentication, and run-owned artifacts. Pretending that a static deployment could perform those operations would weaken the trust boundary. The hosted site is therefore a product showcase and installation entry point; its app route explicitly explains how to run the verifier locally and never asks for a repository upload or account.
The hardest technical problems
Proving failure before success
Generating a regression test after diagnosing a bug is not enough. PatchProof has to prove that the test fails against unchanged production code without losing the test itself.
The implementation separates test-only and production changes, guards the regression stage against production mutations, captures the test diff, restores original production state, reapplies only the accepted regression test, and runs it before patch generation. This is the core of the product promise.
Keeping agent narrative outside the verdict
Codex produces valuable diagnosis, test design, patches, and risk context. But PatchProof's verdict comes only from state-machine invariants and command evidence. I had to preserve this separation across the API, UI, evidence report, historical runs, browser workflow, patch application, and GitHub delivery.
Windows, Git, and Docker boundaries
The project was developed and accepted on Windows, which exposed real issues that a happy-path prototype would miss: Git worktree cleanup, long paths inside generated fixture repositories, cross-account “dubious ownership” protection, Docker Desktop named pipes, process-tree cancellation, port conflicts, and files held open by local tools.
One generated sample was owned by the Codex sandbox account while the product ran as the Windows user. I rejected the easy global safe.directory workaround because it broadens Git trust. PatchProof instead detects affected generated samples and offers an explicit bounded repair that recreates only allowlisted fixture repositories under its own local data directory.
A release run also failed because the development sandbox could not access the user's Docker configuration and named pipe. That failure was recorded as an environment boundary, not hidden as a passing product test. The same complete acceptance was rerun with authorized actual-user Docker access and passed.
Expanding support without weakening the contract
Moving from Node to Python, Go, browser workflows, and custom profiles required more than changing a test command. Each ecosystem has different dependency provisioning, cache ownership, test-file classification, lockfiles, network needs, container images, and cleanup behavior.
The profile architecture made these capabilities explicit. Browser execution remains enabled only for built-in profiles with an accepted browser-server capability. Python dependencies are installed into run-owned caches, and Go uses bounded module download and test commands. Custom profiles are command-locked and mutation-guarded rather than treated as arbitrary shell hooks.
How Codex and GPT-5.6 were used
GPT-5.6 Sol is part of the shipped runtime through the TypeScript Codex SDK. It performs the work that benefits from model reasoning:
- understanding repository context;
- diagnosing the reported defect;
- designing the smallest useful regression test;
- producing a minimal production repair; and
- explaining residual risk.
PatchProof independently performs the work that must be deterministic:
- repository and capability validation;
- worktree isolation;
- baseline execution;
- pre-patch regression reproduction;
- post-patch target verification;
- full-suite verification;
- mutation classification;
- cleanup;
- artifact checksums; and
- the final VERIFIED, UNVERIFIED, or BLOCKED classification.
Codex also accelerated the development process itself. I used it throughout the project rather than only for initial scaffolding. It helped turn the original idea into typed contracts, implement the headless vertical slice, design the state machine, debug Windows and Docker failures, build the browser evidence workflow, add Python and Go profiles, harden custom-profile trust boundaries, write focused tests, inspect screenshots, improve the local UX, prepare clean-clone acceptance, and document decisions and rejected alternatives.
The development log records failures rather than rewriting history after they were fixed: an initially unsupported model identifier, restricted network access during the SDK probe, Docker permission boundaries, a generated fixture that violated Git ownership expectations, a Vercel root-directory deployment failure, transient parallel E2E timeouts, and manual runs that correctly blocked when Docker Desktop was not running.
Important product decisions stayed explicit even when Codex accelerated implementation. I chose command evidence over model narrative, local isolation over direct checkout edits, bounded support over universal claims, explicit confirmation over automatic delivery, and an honest static showcase over a fake hosted verifier.
Product experience and testing
The dashboard includes repository inspection, structured command review, code and browser bug modes, live stages, elapsed time, command evidence, proof conditions, artifacts, persistent history, guarded patch application, GitHub delivery, environment readiness, and five local sample workflows.
The guided Demo view provides the shortest honest evaluation path. It checks the local environment, prepares one bounded Node sample, and prefills the exact bug contract without starting Codex or executing repository commands. The final Start verified fix action remains explicit because it consumes Codex credits and runs repository commands.
The release process includes unit and integration tests, strict TypeScript checking, production builds, dependency audit, tracked-secret scanning, desktop and mobile Playwright coverage, clean-clone rehearsal, and real Docker acceptance workflows for browser, Python, Go, and custom profiles. Those acceptance runs prove the same failure-before-success contract and verify that original checkouts remain unchanged.
The canonical repository includes judge quick-start instructions, supported-platform requirements, sample materialization, manual testing guidance, startup recovery, bounded support claims, and a single release verification command. The public showcase and local dashboard are also tested separately so deployment cannot accidentally expose a nonfunctional verifier UI.
Accomplishments I am proud of
- PatchProof is a working local product, not a mock verification screen.
- The final verdict cannot be awarded by the model that wrote the change.
- The same regression test is proven failing before and passing after the patch.
- Original source checkouts remain isolated unless the user separately approves delivery.
- Evidence remains inspectable through raw logs, commands, reports, diffs, patches, metadata, and checksums.
- Node.js/TypeScript, Python/pytest, bounded Go, guarded custom profiles, and structured Node browser workflows share one proof contract without pretending to be universal.
- The product includes explicit recovery for BLOCKED and UNVERIFIED outcomes instead of presenting every agent change as a success.
- The public deployment accurately explains the local boundary rather than claiming cloud verification it cannot perform.
What I learned
The biggest lesson is that model capability and product support are not the same thing.
GPT-5.6 can reason about Java, Rust, Go, Python, JavaScript, and many other ecosystems. But a trustworthy verifier must also understand how dependencies are installed, how tests are discovered, where caches are written, what files a regression stage may mutate, how commands are isolated, and what cleanup guarantees are possible. Letting the model guess those details and then accepting its own report would erase the reason PatchProof exists.
I also learned that transparent failure is a product feature. A baseline failure, missing Docker engine, invalid generated test, unsupported repository, changed source HEAD, or interrupted live connection should produce an actionable BLOCKED or UNVERIFIED result—not a vague success message.
Finally, the work reinforced that a narrow proof contract can support a broad product direction. PatchProof does not need to promise every language today. It needs a safe adapter boundary that allows each new ecosystem to earn support without weakening what VERIFIED means.
What's next
The next extensions would preserve the same contract while adding selected monorepo layouts, CI execution, signed shareable reports, team policy controls, flaky-test reproduction through repeated or seeded runs, API-contract verification, migration checks, and additional ecosystem adapters.
I would also improve environment gating so a Docker engine that becomes unavailable after the readiness check produces a direct precondition message before dependency or baseline execution. Browser support can expand only after each new runtime has an independently accepted server and replay capability.
The long-term goal is not to make coding agents less capable or less autonomous. It is to give developers, reviewers, and teams a reproducible reason to trust the patches those agents produce.