Inspiration
Despite the fact that AI coding agents can produce patches that appear convincing, follow familiar patterns, and even resemble previous successful fixes, similarity is not proof that the requested behavior has been fixed.
It is possible that a patch may pass the current test suite but fail the one that matters. If it arrives with incomplete or inconsistent provenance, there is no way to determine if the evidence pertains to the patch.
That was the motivation behind DiffProof: Models may propose a fix or an adversarial test but they should not decide if the patch is right. DiffProof splits proposal and judgement. Git provides the exact patch, pytest provides behavioural evidence and deterministic policy provides the final verdict.
What it does
DiffProof is a deterministic behavioural-verification and provenance gate for candidate AI-generated patches. For one very patch it:
- Verifies the exact repo base and patch
- Runs the current test suite on the unchanged base and patched candidate
- Gets one bounded adversarial regression test from a checked-in fixture or an optional model request
- Validates the generated test before execution
- Runs the same adversarial test on both versions
- Checks that the test fails on the base and passes on the candidate
- Validates the supplied provenance metadata
- Produces three independent results:
- Behaviour: VERIFIED, BLOCKED or INCONCLUSIVE
- Provenance: COMPLETE, PARTIAL, MISMATCHED or INVALID
- Composite gate: PASS, BLOCK or REVIEW DiffProof has also:
- Deterministic multi-run replay analysis
- A strict fixture-only verify_patch programmatic tool
- A strict fixture-only verify_patch function-calling adapter that makes no network request
- An offline GitHub pull-request dry run
- Deterministic PR review Markdown and suggested check conclusions
- A read-only localhost evidence dashboard
- Sanitised judge-facing examples for verified, blocked, replay, tool, and PR outcomes The dashboard shows only saved evidence. It never calls git, tests, patches, openai, codex or github.
How I built it
Codex, running on GPT-5.6, was a collaborator in DiffProof's staged development writing modules, tests, and fixes. AGENTS.md and SPEC.md directed that process and every change was gated by pytest, ruff and strict mypy. GPT-5.6 inside the product is only allowed to suggest one structured adversarial test, and codex cli is the optional live replay patch runner, and neither is allowed to decide verdict, gate, replay score or PR conclusion. I wrote DiffProof in Python 3.12+ with Pydantic for strict evidence contracts, git for patch authority and pytest/junit output for behavioural evidence. The core verification loop makes isolated git workspaces for the base and candidate. Inside made commands run with fixed argument vectors, shell=False, timeouts and sanitized environment. The adversarial test that has been generated is piped into an ast-based validation gate before it can run. The same accepted test source is run against both workspaces. DiffProof does not trust model explanations or confidence scores, verdict is computed from saved execution evidence. Provenance is handled separately. DiffProof records and checks declared base, patch and task hashes without claiming external authenticated identity or line-level authorship. I then added several integration layers without duplicating the verifier: The programmatic tool uses a strict checked-in json schema and host-controlled repo and output roots. The github pr adapter validates exact diff-base SHA, head SHA and git-authoritative canonical diff before calling the existing verifier. The replay system classifies repeated runs as STABLE, MIXED, UNSTABLE, INSUFFICIENT. The local dashboard is a python standard-library http server, fixed read-only routes, typed artifact loaders, escaped html, embedded css and ipv4 loopback-only binding. Everything is made so that integrations transport and display evidence and never become new decision authorities.
Challenges I ran into
The hardest problem was to distinguish a correct-looking patch from a patch that has some behavioural evidence. The existing tests were not enough and I needed a deterministic oracle that the adversarial test fails on the original code and passes on the candidate. The Windows workspace behaviour was another hard problem. Live Codex replay showed transient sandbox and workspace-layout problems. I solved it with self-contained local clones, exact base verification, layout preflight checks and fail-closed execution states. The GitHub pull-request verification needed careful diff semantics. The target branch tip is not always the commit the PR patch was generated against. DiffProof uses an explicit diff-base commit and the supplied patch has to agree with Git's canonical base-to-head diff. Security boundaries posed another problem. The repository name, PR content, tool arguments and artifact references were all treated as untrusted input. I used canonical path containment, traversal protection, symlink/junction/reparse rejection, bounded JSON parsing, duplicate-key rejection, sanitized errors and relative artifact references. The last problem was evidence packaging. The Live replay data contains paths, session metadata and raw events, so I made separate sanitized examples and aggregate summaries instead of publishing private execution transcripts.
Accomplishments that I'm proud of
I am most happy that DiffProof has a clear authority boundary: The model proposes. Git, pytest and deterministic policy decide. The real live replay acceptance completed three Codex patch runs: 3 requested runs 3 completed patches 3 VERIFIED results 3 unique patches 100% replay score 0 execution failures 0 no-patch runs I am happy about the separation of behavioural validity and provenance. A patch can be behaviourally verified while still requiring human review because its provenance is not complete. But a complete provenance cannot save a patch that fails behavioural verification. Other things I am happy about are: strict immutable evidence models; deterministic and explainable reason codes; reproducible offline demos; fixture-only programmatic tool calling; exact GitHub PR base/head/diff validation; injection-safe PR review output; a secure, dependency-light evidence dashboard; a comprehensive automated test and clean-checkout validation suite; judge-facing documentation, screenshots, architecture and evidence packages.
What I have learned
The most important thing I have learned is that evidence is more important than confidence. A model can explain a patch well and still be wrong. A deterministic test that fails before the patch and passes after it gives a much stronger signal. I also learned that behaviour and provenance must be judged separately. They are answering different questions: Did the patch fix the requested behaviour? Does the given origin information always describe this patch? Integration layers must stay transport boundaries. A GitHub adapter, tool-calling interface or dashboard should not quietly add its own verdict rules. I also learned to be more precise about security claims. DiffProof is for trusted local repositories and tests. AST validation reduces risk but is not a hostile-code sandbox. Supplied provenance improves traceability but is not external identity authentication or proof of line-level authorship.
What's next for DiffProof
The next step is to add optional live GitHub transport behind the existing injected reader and writer boundaries. That would allow organisations to publish check runs or PR comments without changing the deterministic verifier. Other future improvements are: externally authenticated provenance attestations; bounded streaming of subprocess output; broader language and test-framework adapters; support for larger statistically meaningful replay studies; more adversarial-test generators; policy configuration for different repository risk levels; CI integrations that use the existing structured result format; richer evidence comparison of multiple candidate patches. The long-term goal is for DiffProof to be a trusted verification layer between AI coding agents and software delivery systems without the same model that proposes a change being the authority that approves it.
Log in or sign up for Devpost to join the conversation.