ProofRead

ProofRead’s scripted offline demo records real code edits and test executions, showing verified claims alongside checks.

Inspiration Coding agents can sound convincing even when a test failed, a command timed out, or a check never ran. I wanted a way to answer a simple question: What evidence supports the agent’s claim that it finished? That led to ProofRead: a coding agent where the model proposes actions and an independent verifier determines what the recorded results actually support.

What it does ProofRead inspects a repository, makes code changes, runs checks, and records the results in an evidence ledger. It evaluates completion claims as verified, partially verified, failed, or unverified. Through a local dashboard, developers can inspect each claim, trace it to tool outputs, review code changes, and download a portable Proof Bundle containing reports, evidence, and diffs. Bundles can be reviewed and validated offline without the original repository. Reviewers can also set acceptance requirements, such as passing tests or a verified build. The Proof Score summarizes observed verification coverage; it does not represent the probability that the code is correct.

How we built it I built ProofRead in Python, using FastAPI for the backend, Streamlit for the interface, and SQLite for append-only execution records. NVIDIA Nemotron, accessed through Nebius Token Factory, powers the live agent’s repository inspection, tool selection, and code generation. I used prompt engineering and structured tool calls without fine-tuning the model. A separate deterministic verifier evaluates completion claims using command exit codes, JUnit test results, file hashes, repository fingerprints, and validated Python wheel metadata. Unsupported or failed claims are returned to the agent for further verification or removal within a bounded loop. Each run can export a ZIP bundle containing execution evidence, verification history, and the final patch. Offline validation checks SHA-256 hashes, consistency between files, and whether recorded verdicts can be reproduced from the bundled evidence.

Challenges we ran into The hardest challenge was deciding when evidence still applies. A passing test result becomes stale after another code change. Tests can even modify source files while running. I added repository fingerprints and before-and-after observations to tie checks to the state they actually examined. I also had to separate a completed agent run from a successful repair. A run can finish while its claims remain failed or unsupported. Preserving rejected claims and verification history makes those outcomes visible. Another challenge was making exported evidence useful outside the application. I built bundle validation and offline review so someone else can inspect a run without the original repository or rerunning its code.

Accomplishments that we're proud of I’m proud of completing a genuine live Nemotron repair in which the agent inspected a pricing repository, reproduced a bug, changed the code, and moved the test results from three passes and one failure to four passes. The run took seven model requests, and its exported bundle passed validation and the configured acceptance policy, with a Proof Score of 90/100. I also built five controlled offline scenarios covering a successful repair, an incorrect patch, a timeout, unsupported claims, and Python wheel packaging. These scenarios demonstrate how ProofRead handles both successful and unsuccessful outcomes. The offline scenarios use scripted decisions with real local execution. The live repair used genuine model responses and represents one successful fixture rather than a broad reliability benchmark.

What we learned I learned that reliable coding assistance needs a clear connection between a claim, an execution result, and the repository state that was checked. A convincing final response alone is not enough to establish that a task succeeded. Missing evidence also deserves its own status. “Unverified” means something different from “failed,” and passing tests support a narrower conclusion than proving a bug is completely fixed. I also learned to distinguish integrity from authenticity. Hashes can expose corruption and inconsistent records, but unsigned bundles cannot establish who produced the evidence. That distinction shaped how I describe ProofRead’s guarantees.

What's next for ProofRead I want to evaluate ProofRead across more real coding tasks, add isolated execution environments, support more build systems, and strengthen evidence authenticity. I also plan to improve the review experience so developers can understand failed and unsupported claims more quickly, and explore a hosted version. The current prototype runs locally, with paid inference disabled by default.