From 500 open PRs to 5 maintainer decisions. Fast screening, clustering, and adversarial testing progressively narrow the queue.
Five decisions, not five hundred PRs. Pull Guard turns a noisy queue into clear, evidence-backed actions for maintainers.
Merge order matters. Pull Guard tests interacting PRs together and identifies the safest sequence before they reach the main branch.
PR floods surface as one decision. Near-identical submissions are detected, grouped, and reviewed as a single wave.
One change, many attempts. Pull Guard clusters repeated solutions and routes each group toward one representative review target.

About Pull Guard

Inspiration

AI coding agents have made code cheap to produce. A repository that used to get a dozen pull requests a week now gets hundreds. Many of them are near-identical attempts at the same fix, many are automated dependency floods, and most arrive with a confident description that claims more than the diff actually does.

Maintainers now spend most of their review time figuring out which PRs are even worth reading. The bottleneck in open source is no longer writing code. It is verifying it. That is the problem Pull Guard addresses: a pull request description is a claim, not a fact, so we built a system that checks the claims before a human has to.

What it does

Pull Guard sits between a pull request and a merge. It fingerprints every incoming PR, deduplicates overlapping attempts into a single comparison, detects flood waves of near-identical submissions, and filters out low-quality or policy-violating work early. For the PRs that survive screening, Codex generates adversarial tests based on the claims in the description, and those tests run in disposable sandboxes and get graded. Pull Guard also tests PRs in pairs to find the safest merge order. The result is a funnel:

$$500 \text{ open PRs} \;\rightarrow\; 60 \text{ candidates} \;\rightarrow\; 15 \text{ review targets} \;\rightarrow\; 5 \text{ decisions}$$

Every final decision (review first, safe to merge, request changes, superseded, or a tested merge order) comes with its evidence attached. Pull Guard never merges or closes anything by itself. It recommends, humans decide.

How we built it

Fingerprinting and immutable versions. Every PR is reduced to a canonical, patch-complete fingerprint: a SHA-256 hash over the full patch content of every changed file, plus filenames and renames. These fingerprints are the identity layer for the whole system. They drive cache keys and idempotency, so re-analyzing the same patch is free, and a force-push that changes one line anywhere produces a new identity and invalidates nothing else. Each analyzed state of a PR is stored as an immutable version record, which means every piece of evidence we produce can be traced back to the exact code it was produced from.

Clustering and deduplication. Clustering works on two levels. First, structural overlap: files, symbols, and patch hashes are compared across the queue to find PRs touching the same subsystem. Second, semantic intent: the GPT-5.6 family reads each diff and its description and assesses whether two PRs are attempting the same change, even when the implementations look nothing alike. On top of that sits a supersession detector that classifies redundant pairs as exact duplicates, subsumed implementations (one PR does everything the other does, plus more), weaker test coverage on the same target, or partially obsoleted subsets. Detection is sticky: the evidence hash is recorded with the pair, so a force-push does not silently clear a redundancy finding, and the record auto-resolves when the canonical PR merges. Nothing is auto-closed; the maintainer gets one comparison instead of seven tabs.

Claim extraction and adversarial tests. This is where Codex does the heavy lifting. First, a claim-extraction pass reads the immutable PR context and pulls out every verifiable claim the description makes, with strict provenance: each claim has to cite the source it came from, and a grounded retry pass handles vague or ungrounded output. Then, for each selected claim, Codex generates a candidate adversarial test designed to break that claim. Generated code never touches the control-plane checkout. It is stored as an immutable artifact and only applied inside an isolated runner worktree after the patch and command pass validation. Runners are disposable Docker containers started with --network none, empty required environments, and restricted mounts, and every execution is followed by an isolation audit that verifies the hardening was actually applied rather than merely planned. Grading is strict: a proof only counts if the test fails on the base code and passes on the PR, re-run across fresh sandboxes to catch flakiness. This is the part of the system that the GPT-5.6 family makes economically possible. Writing tests like this used to be senior-engineer work done under time pressure. Now it is seconds per claim, at a cost low enough to run against an entire queue, and the tests are consistently better targeted than what we wrote by hand.

Merge order and interaction testing. Merging is sequencing, so Pull Guard treats the queue as a graph. For pairs of related PRs it schedules durable, ordered interaction runs: apply A then B, and B then A, in isolated worktrees, and record a compatibility verdict, the conflict stage if one fails, verification exit codes, and timing. The maintainer-facing Queue Plan is a read-only projection of that graph, so the safest path to main is visible at a glance. When a cluster genuinely needs changes from more than one PR, composite runs validate selected immutable versions together and repair real conflicts mechanically, again without writing to GitHub. Rebase flags surface stale PRs that will rot before they merge. Because pairwise testing grows quadratically, the decision funnel deliberately spends this scarce proof capacity only where the result can change a maintainer's decision.

Control plane: Python/FastAPI backend, durable analysis queue with outbox events, runner recovery, stage-level progress tracking.
Frontend: React + Vite dashboard with a Command Center, cluster comparison, decision workspace, Queue Plan, and funnel analytics.

Challenges

Runner reliability: sandboxes crash mid-test. We built queue recovery and retry semantics so a failed runner never silently loses evidence.
Test quality: early generated tests were too easy to pass. Requiring fail-on-base and pass-on-PR, across fresh runs, is what makes a proof a proof.
Interaction explosion: pairwise merge-order testing is $O(n^2)$; the funnel had to shrink $n$ aggressively before pairs were ever considered.
Trust boundaries: every action stays a recommendation. Pull Guard never writes to GitHub.

What’s next

We plan to wrap Pull Guard’s verification pipeline in an agent-facing CLI. Coding agents will be able to run it against their own patches before opening a pull request, receive structured feedback on unsupported claims, failing adversarial tests, redundant implementations, and merge interactions, then revise their work and retry without waiting for human review.

The goal is to turn Pull Guard into a verification loop agents can use autonomously: generate, prove, repair, and only submit once the patch meets the repository’s standards. Machine-readable output, stable exit codes, immutable run references, and configurable quality gates will make it usable from agent harnesses and CI systems without depending on the dashboard. Actions such as merging or closing pull requests will remain separately permissioned.

What we learned

When generation is cheap, verification becomes the expensive part, and verification only scales when it is staged: cheap filters first, deep proofs last. We also learned how much the frontier models changed what a small team can build. The GPT-5.6 family does real analytical work in this system: reading diffs, extracting claims with provenance, judging semantic intent, and writing adversarial tests that survive strict grading. Codex is not a convenience feature in Pull Guard; it is the reason per-PR proof is feasible at all. Finally, maintainers only trust an AI review system when the trust is structural: immutable versions, reproducible runs, visible evidence, and a strict rule that the system advises but never acts.