Exhibit A

An evidence engine for code that is only allowed to speak with proof.

Comment

Inspiration

AI code reviewers have a trust problem. They cry wolf. A reviewer that flags ten issues with eight false positives trains developers to ignore all ten. Exhibit A starts from a stricter rule: a runnable fail-to-pass test, or silence.

What it does

Exhibit A is an evidence engine for code. Codex reads a bug claim and a repository in a read-only sandbox. It proposes a focused pytest reproduction and refines it from execution feedback. The engine runs only that generated test against the reported state and the fixed state.

A deterministic flip check is the sole judge. A Case is PROVEN only when the test fails on the reported state for the expected reason, passes on the fix, remains deterministic across reruns, and clears tamper, infrastructure, and scope checks. If no candidate clears every gate, Exhibit A records INSUFFICIENT_EVIDENCE and stays silent.

Detective mode turns a bug report into a verified reproduction. Prosecutor mode allows a review comment only when execution proves the behavior change. The web interface streams each attempt over Server-Sent Events, so the user can watch the model try, fail, refine, and either prove the claim or decline to speak.

How we built it

The Python evidence engine separates the model from the judge. Codex with GPT-5.6 Sol handles localization, planning, test drafting, inversion, and bounded refinement. It never decides the verdict. The model output is treated as untrusted input.

The executor runs against disposable repository copies. Docker execution disables the network, drops capabilities, uses a read-only root filesystem, and enforces resource limits. Remote repository intake accepts HTTPS URLs and validated commit SHAs. Git hooks are disabled, and commands receive untrusted values as argv without shell interpolation.

The Next.js and React interface drives the same engine through a streaming API. It includes the Detective workflow, the Prosecutor evidence gate, deterministic Case replay, the Silence Ledger, and the false-conviction self-audit.

Open science

Every PROVEN Case contains an execution-validated test tied to concrete code states. That makes it useful as a reproducible AI-for-software-engineering research artifact. Exhibit A includes signed Executable Evidence Format bundles, mutation scores, verified minimization, evidence strength measurements, reproduction convergence studies, oracle-gap probes, environment setup records, execution-based bug deduplication, release-history archaeology, counterpatch triangulation, and property escalation.

These research measurements are descriptive. They never override the flip check.

Challenges

The hardest problem was preserving a sharp trust boundary while still using a capable coding agent. Generated code, repository contents, claims, URLs, SHAs, and patches all had to remain untrusted. Another challenge was making honest silence feel like a product outcome rather than an error state.

Accomplishments

The repository contains a working engine and product interface with 132 passing engine tests, typed web tests, clean lint and formatting checks, and a production web build. Local checkout intake, two-SHA Git intake, live execution logs, deterministic replay, silence recording, and the research instrumentation are implemented.

How Codex and GPT-5.6 were used

Codex with GPT-5.6 Sol served two roles. It was the implementation partner for the engine, security boundaries, test suite, documentation, and streaming interface. It is also a product component. Inside Exhibit A, Codex proposes reproductions from a read-only sandbox while the deterministic engine independently executes and judges them.

This division mirrors the product thesis. The model reasons, but only execution is allowed to speak.

What is next

The next step is evaluation on a larger set of real bug-fix pull requests. The project can measure false convictions, environment build rates, reproduction convergence, and benchmark oracle gaps without turning those measurements into unsupported claims.

Exhibit A can also evolve toward a provider-neutral hypothesis layer. Explicit adapters can support configured hosted APIs and opt-in discovery of user-approved local model runtimes, so generation is not tied to a single source. Credentials will stay in environment variables or operating system credential stores and out of Case files, prompts, and logs. Every provider will remain an untrusted test proposer, and the deterministic flip check will remain the sole judge.

Built With

  • ai
  • ai4se
  • automated-testing
  • code-review
  • codex
  • docker
  • git
  • github-actions
  • gpt-5.6
  • llm
  • next.js
  • openai
  • pytest
  • python
  • react
  • ruff
  • server-sent-events
  • tailwind-css
  • typescript
  • vitest
Share this project:

Updates