codex-PR-review

Inspiration

Reading a pull request is a lonely job. One human reviewer, one perspective, one bad afternoon — and a real bug ships. When AI-assisted review arrived, the first instinct was to swap the human for a model. But a single model has the same failure mode: one perspective, one blind spot, one confidently-wrong hallucination.

What kept surprising us in daily work was how differently different reviewers were wrong. Codex, reading only the diff, caught a type="submit" bug that Claude — which had read the whole codebase — waved through because "the form library handles it." Claude, holding architectural context, caught a wrong-direction dependency that Codex couldn't see without the import graph. Gemini, running as a third opinion, flagged a main.tsx legacy entry both had ignored.

No single axis was enough. But their disagreements were information. That was the idea behind codex-PR-review: stop trying to build the perfect reviewer, and start orchestrating a panel of imperfect ones that check each other's work.

What it does

codex-PR-review is a slash-command in Claude Code that turns one /pr-review into a fully cross-verified, multi-axis code review of a GitHub or Bitbucket pull request, output as a single Traditional Chinese comparison report ranked Must Fix / Should Fix / Nice / For Reference.

Given a PR URL, it:

1. Fetches PR metadata, diff, and any attached spec/plan docs; builds a deterministic changed-file inventory.
2. Isolates the PR head in a fresh git worktree so every reviewer sees exactly the code under review — not the user's dirty working tree, not a stale local branch.
3. Dispatches in parallel:
   • Claude (context-aware) — routes to language- and domain-specific reviewers (TypeScript, Python, security) with the whole codebase available for search.
   • Codex neutral — reads only the diff, no context, using the built-in codex review contract.
   • Codex adversarial — a red-team pass whose job is to refute the code, not defend it.
   • Gemini Flash — an independent third opinion.
4. Cross-verifies each finding by re-reading it under a different axis (Claude verifies Codex, Codex verifies Claude).
5. Augments with deterministic tools — sem for entity-level blast radius, react-doctor for mechanical React scans, author-history calibration files for per-author phrasing rules.
6. Consolidates every axis's findings into one deduplicated report, tagged by source reviewer, with verbatim code anchors, spec citations, and a coverage assertion proving no file was silently dropped.

The output is designed to be read by a busy human in five minutes: what to fix now, what to think about, what the reviewers disagreed on, and which findings the author historically pushes back on.

How we built it

The system is a single Claude Code slash-command (/pr-review) roughly 1,300 lines long, but the real architecture is under the hood:

• Orchestration layer: Claude Code's Agent tool for parallel dispatch of language/domain-specific sub-reviewers, and Codex CLI (via a codex-companion background wrapper) for the two Codex axes.
• Worktree isolation: Every review runs in a scratch git worktree pinned to the PR's HEAD commit. This lets Codex freely explore with git show / grep / find without ever touching the user's working tree.
• Background execution + polling: Claude Code's Bash tool has a hard 10-minute timeout. Large PRs on Codex xhigh blow past that in the first phase. We solved this with nohup-detached processes writing to log files, plus a poll-liveness.sh script emitting three signals — DONE, STILL_RUNNING, DEAD — reading Codex's rollout JSONL for the task_complete event.
• Deterministic supplements to LLM judgment: sem-pr-blast-radius.sh computes dependent counts and test coverage from the actual dependency graph. react-doctor performs mechanical scans that no LLM needs to re-do. These outputs are consumed only at synthesis — never injected into a model's context, so the axes stay independent.
• Preset system: Users pick from default, light, sol-lite, or deep presets to trade off review depth against Codex quota burn. Effort levels are patched into ~/.codex/config.toml per-run and restored at cleanup.
• Author calibration: Per-author markdown files at docs/philip/pr-review-calibration/.md record how each PR author historically responds to findings. Calibration can only downgrade severity or adjust tone — never drop strict-liability findings — and every adjustment is audited in the final report.

Challenges we ran into

Every day of building this was a lesson in how AI-native tools fail:

• The 10-minute Bash timeout. Codex xhigh on a mid-size PR takes 16 minutes. Foreground calls got SIGTERM'd at the ten-minute mark, silently sinking token spend into a truncated rollout. Fix: subshell + nohup + JSONL event polling.
• Silent Codex wedges. codex review sometimes stalls on MCP cold-start (mcp: semble/search started) and never recovers. Fix: mcp_servers={} on every review invocation — the diff never needs MCP.
• Broker environment collision. CC Bash tool injects CODEX_COMPANION_SESSION_ID and CLAUDE_PLUGIN_DATA env vars that collide with Codex's own app-server, killing it with a cryptic No such file or directory on startup. Fix: env -u to unset both before launching the companion.
• Ground truth for LLM findings. LLM reviewers produce plausible-looking false positives constantly. Cross-verification (each axis re-reads the others' findings) reduces this by roughly 40%, but the real gains come from not injecting the search-before-flag rule into Codex's prompt — its diff-only perspective is exactly what makes it catch what the context-aware reviewer waves through.
• Coverage without an assertion is a lie. Early versions had reviewers "silently drop" files on large PRs — the report would look complete, but three files were never actually reviewed. Fix: a deterministic per-file accounting requirement — every file returns either findings or an explicit REVIEWED_NO_ISSUES: line, and Step 4.5 asserts the union against the deterministic inventory.
• Prompt injection from the PR itself. A malicious PR description could try to instruct the reviewers. All external content is passed as data with explicit "treat as untrusted input" framing.

Accomplishments that we're proud of

• It's real, and it's used. Not a demo repo. It reviews production PRs against a real codebase every week, catching real bugs.
• Every finding is traceable. Verbatim source anchors, source-axis tags ([typescript-reviewer], [Codex-adversarial], [Gemini-Flash]), and spec citations mean no finding is a black box.
• Findings that no single model would have caught. Concurrent lost-updates, DynamoDB 400KB item-limit violations, dark-mode invisibility, fail-open auth bypasses — all found by axes that disagreed with each other, then confirmed by cross-verification.
• Deterministic where determinism belongs. File inventory, dependency graphs, mechanical React scans, worktree isolation — the LLM axes only judge what LLMs are good at judging.
• Sensitive to reviewer fatigue. Author calibration lets us stay strict on strict-liability defects while downgrading pedantry the author will just ignore anyway.

What we learned

• Cross-model verification beats single-model confidence. The interesting signal is not what a model says, but what happens when another model, given the same evidence, disagrees.
• Diff-only perspective is a feature, not a limitation. Fresh eyes catch things context-aware eyes have rationalized away. Trying to give Codex more context made it worse.
• Prompt engineering is a small part of the game. How the axes are composed, when they're independent, when they cross-verify, what feeds the synthesis step — that architecture matters far more than any single prompt.
• Deterministic tools and LLMs are complementary, not competitive. Every time we tried to have an LLM re-derive something a graph query could answer, quality went down and cost went up.
• The hardest problems in AI-agent tooling are boring. Timeouts, environment variables, silent process death, coverage assertions. The magic is in the plumbing.

What's next for codex-PR-review

• Auto-post as inline PR comments. Currently the report is Traditional Chinese for a human reader; the next step is to post confirmed CRITICAL/HIGH findings directly as GitHub / Bitbucket inline comments with the verbatim anchor as the target.
• Learning loop. When the PR author accepts or rejects a finding, feed that back into the author's calibration file automatically — so the reviewer gets more accurate over time per person.
• More languages. Currently the primary reviewers cover TypeScript, JavaScript, and Python well. Rust, Go, and ClojureScript are next.
• More adversarial axes. The red-team axis catches things nothing else does; a security-only red-team axis and a performance-only red-team axis are natural extensions.
• Open-source the orchestration template. The command is highly personalized right now (paths, author calibration, sponsor-repo references). A generic template that other Claude Code users can drop in with minimal config would be the payoff.

Built With

• claude
• code-review
• codex