Inspiration
AI coding agents can generate convincing patches quickly, but a plausible-looking diff is not the same as a verified solution. Agents may misunderstand requirements, modify tests to make failures disappear, repeat ineffective edits, or consume resources indefinitely.
We built BenchProof to make autonomous coding safer and easier to review. Instead of asking users to trust an agent’s final answer, BenchProof records the plan, code changes, test results, resource usage, and execution history that produced it.
What it does
BenchProof is a task-centered autonomous coding agent powered by NVIDIA Nemotron through Nebius Token Factory.
A user imports a Python project and describes a bug or feature. BenchProof then:
- Safely inspects the project and creates a fingerprinted source snapshot.
- Generates a step-by-step implementation plan.
- Waits for explicit human approval before changing code.
- Protects imported tests from modification.
- Generates a scoped patch using only approved editable files.
- Runs the candidate in an isolated Docker sandbox with no network access and strict time, memory, CPU, and process limits.
- Uses real compiler and test feedback for bounded revision attempts.
- Stops safely when it detects repeated patches, exhausted budgets, invalid output, or infrastructure failures.
- Presents the final diff, test output, checkpoints, hashes, and execution trace for human review.
The result is a review-ready evidence bundle—not an automatic claim that every requirement has been satisfied.
How we built it
We use NVIDIA Nemotron-3-Nano-30B-A3B, served through Nebius Token Factory, for planning, diagnosis, and code generation.
The backend is built with Python 3.12, FastAPI, Pydantic, and SQLite. A durable worker processes coding tasks using leases and fencing tokens so interrupted jobs can recover without accidentally executing the same operation twice.
Candidate code runs inside isolated Docker containers. The runtime denies network access, uses a non-root user, mounts protected source and test assets as read-only, and enforces resource and execution limits.
The frontend uses React 19, TypeScript, and Vite. It provides a task workspace with plan approval, an execution timeline, budget gauges, test output, a unified diff viewer, checkpoint controls, and downloadable evidence.
We also added bounded customization through instruction-only skills, saved profiles, reviewed test-harness policies, and a consent-based read-only documentation connection.
Challenges we faced
One major challenge was preventing the agent from appearing successful by weakening or changing tests. We solved this by separating editable source files from protected evaluator assets and validating every generated patch before execution.
Another challenge was handling model failures safely. Generated output can be malformed, incomplete, repeated, or truncated. BenchProof validates structured responses, limits revision attempts, detects duplicate candidates, and records a precise stop reason instead of retrying forever.
Cloud execution also introduced worker crashes, timeouts, stale jobs, and concurrent updates. We implemented durable checkpoints, leases, compare-and-swap state transitions, admission controls, and recovery tests for these cases.
Accomplishments that we are proud of
- Built a complete plan–approve–edit–test–review workflow.
- Integrated real NVIDIA Nemotron inference through Nebius Token Factory.
- Executed generated code in isolated, resource-limited Docker environments.
- Completed a live bug-fix demonstration in which a failing baseline became a review-ready patch with all supplied checks passing.
- Added rollback checkpoints and sanitized evidence exports.
- Created a responsive desktop and mobile review experience.
- Preserved failed and blocked runs alongside successful ones instead of hiding them.
- Built extensive backend, frontend, deployment, security, and cloud-failure regression coverage.
What we learned
Passing tests is evidence, but it is not proof that every unstated requirement is satisfied. Reliable coding agents need explicit boundaries, human approval, fresh execution, limited retries, and honest failure reporting.
We also learned that provenance matters. A reviewer should be able to connect an approved plan to the exact source version, patch, execution, and result being presented.
What's next
We plan to add support for TypeScript, Go, and Rust; Git branch and pull-request integration; larger parallel evaluation pools; richer policy controls; and broader held-out evaluations of repair quality.
Our long-term goal is to make AI-generated code changes easier to inspect, reproduce, and trust.
Log in or sign up for Devpost to join the conversation.