Cross-Examine

The problem

Coding agents can now produce more code than any team can meaningfully review. And the tests pass — because the agent wrote the implementation and the tests that cover it. Nothing in that loop checks whether the behavior the diff replaced still holds.

So the model fixes one bug, introduces another, and the suite stays green the whole time. Most AI review tools answer this with a second model opinion, which leaves the same kind of system that wrote the code responsible for declaring it safe.

We wanted an independent witness: a process that treats model output as a hypothesis, executes the code, and refuses to claim safety without evidence.

What it does

Cross-Examine resolves two Git revisions into separate detached worktrees and catalogues candidate definitions in changed Python files. GPT-5.6 reads bounded diff and source context and emits schema-constrained Claims — proposals about behavior that must be preserved. The schema has no field for an outcome, so the model structurally cannot return a verdict.

Then execution takes over:

Layer A captures how the base revision actually behaves, and replays the identical inputs against head.
Layer B runs 60 derandomized Hypothesis examples with shrinking, hunting counterexamples the claim never mentioned.
Repository tests run against both revisions, so pre-existing setup failures are never misreported as regressions.
aggregate() — 16 lines, pure, no I/O, no network — maps findings to SAFE, RISKY, or BROKEN.

Every decided finding opens to its receipt: the exact argument vector executed, the raw captured output from both revisions, and the input that reproduces it.

The offline hero demonstrates a plausible optimization that returns None for an empty list while its happy-path test stays green. Cross-Examine returns BROKEN with [] as the reproducing input — in about two seconds.

How we built it

Python, FastAPI, SQLite, React, Vite, and Tailwind. Bounded host subprocess execution, Git worktrees, Hypothesis, and the OpenAI Responses API with structured Pydantic output.

Codex built this system. The five-stage pipeline, the schema and validation layer, the subprocess execution controls, SQLite persistence, the FastAPI service with SSE stage progress, the React evidence explorer, the CLI, packaging, and the cross-platform verification scripts — with 138 Python tests and 32 frontend tests behind them, running on Windows, macOS, and Ubuntu.

The human retained product authority throughout, and that split is why the verdict is trustworthy. Every doctrine on the left constrains what the code on the right may conclude:

Human-provided doctrine	Codex-chosen implementation
Problem selection and Python-only scope	FastAPI / SQLite / React stack
The contract and five-stage structure	Worktree and subprocess mechanics
Abstain-toward-risk policy	Edge catalog and Hypothesis bounds
Layer-A-before-Layer-B sequencing	Persistence and SSE protocol
Trusted-input execution boundary	CLI surface and deterministic hero construction
Evidence doctrine and submission story	Cross-platform diagnosis, release verification

Challenges

The hardest boundary was preventing plausible model output from becoming an oracle. It is genuinely tempting to let a confident-sounding claim stand in for a check. We solved it structurally rather than by prompting: proposals are validated against catalogued candidates, and outcomes are assigned only by deterministic code.

Real-repository trials against unmodified public Python projects exposed three defects in Cross-Examine itself — and Codex found all three:

Child Python inherited Windows cp1252 even when the parent decoded pipes as UTF-8, so generated Unicode could fail before evidence was emitted. UTF-8 child stdio is now forced at the execution boundary.
A raw pytest failure count can be caused entirely by missing optional dependencies. Both revisions now run, and only a passing-base regression can refute.
pytest's cache provider could hit a Windows rename denial inside a detached worktree, so conservative commands disable that non-evidentiary cache.

Each became execution policy with regression coverage, rather than a hidden demo caveat.

Executing a repository is also a security boundary. Commands use argument vectors with shell=False, an executable allowlist, a minimal child environment that strips secret-shaped variables, deadlines, a 2 MB output cap, and receipt redaction.

What we learned

Trustworthy agentic development needs separation of duties. Models are genuinely good at identifying what deserves scrutiny — but execution must own the evidence, and deterministic policy must own the verdict.

We also learned that an honest UNVERIFIABLE is a product feature. Setup failures and unsupported behavior should be visible, not quietly converted into confidence.

Scope

SAFE means bounded, not proven: nothing was refuted among the checks that actually ran — not that the pull request is correct. Cross-Examine executes the target repository's code, so point it only at repositories you trust; production use needs real isolation.

The public evidence explorer serves a labeled, checked-in fixture so the report UI and its receipts can be inspected without installing anything. Executing a repository needs Git and a local runtime, so that path runs locally by design.

What's next

Disposable, network-restricted target isolation so untrusted repositories can be verified safely. A deterministic setup contract so dependency installation becomes part of the evidence. Corpus v2 with Git ancestry rather than locator replay. And intended-change oracles, so a claim about new behavior can be executed instead of abstained on. Inspiration