LoopBench

• 

  LoopBench Infographic

• 

  Agent Evolution Graph Tree

• 

  LoopBench

• 

  LoopBench Tree

Inspiration

AI agents can solve difficult tasks, yet their mistakes often disappear into chat histories. Running more agents creates more attempts, but it does not automatically preserve what worked or explain why an approach failed.

Our idea: a tree for AI agent evolution, where mistakes become permanent version history that informs the next experiment.

What it does

LoopBench is an experimentation platform that helps AI agent harnesses improve through recorded mistakes.

Users connect their Nebius API key, select a model and benchmark, and start a Pi agent experiment. LoopBench measures a baseline, then uses NVIDIA Nemotron to diagnose failures and propose competing harness improvements.

Each evolution round produces 2–6 distinct candidates. LoopBench evaluates each candidate against a freshly rerun parent under the same benchmark and model settings. Improvements are accepted; ties and regressions remain visible with their evidence.

The horizontal tree lets users inspect each version, understand what changed and follow its descendants. Previous outcomes inform later proposals. Evolution continues across rounds until paused by the user, provider availability or resource constraints.

LoopBench changes the agent’s harness instructions and procedures. It does not retrain the model’s weights.

How we built it

We built the interface with Next.js, React and TypeScript, with SQLite storing campaigns, execution checkpoints, comparisons and version history.

NVIDIA Nemotron plays two roles: it powers the Pi coding agent and acts as the optimizer that proposes improvements to that same agent’s harness.

Nebius Token Factory provides inference for both roles. Our gateway authorizes requests, applies inference allowances and records token usage. The model selector uses live provider discovery.

Token Factory Sandboxes provide isolated execution. We prepared a pinned Pi and Node.js runtime image so experiments can start from a consistent environment. Agent environments run without external networking; a trusted host relay carries inference requests to our gateway without exposing the real Nebius key inside the guest.

Separate verifier environments evaluate generated solutions against private acceptance tests. This separates the agent’s work from the system deciding whether that work passes.

We initially integrated E2B, then added Token Factory Sandboxes after receiving beta access. Both remain supported execution providers.

Challenges we ran into

The hardest problem was distinguishing a real improvement from randomness or infrastructure failure.

A candidate scoring higher than an old result is insufficient evidence. We introduced fresh parent reruns and paired comparisons. Infrastructure failures remain unscored rather than becoming artificial losses.

We also had to keep evolution progressing after rejected candidates, recover durable checkpoints and distinguish active evaluation from a genuinely stalled campaign.

Another challenge was visibility. A parent verification can be running before a new branch appears. We added current-round, evaluation-phase and task-progress indicators directly above the graph.

Accomplishments that we're proud of

• Built real execution-backed trees without seeded experiment results.
• Used Nemotron to improve a harness powered by Nemotron itself.
• Executed Pi and independent verification through Token Factory Sandboxes.
• Recorded an accepted candidate passing 2 of 6 tasks versus its freshly rerun parent’s 1 of 6.
• Preserved rejected branches and their evidence for subsequent experiments.

That measured gain is an early result on a small benchmark, not proof of general coding improvement.

What we learned

More compute increases the number of attempts. Improvement requires an additional loop: preserve evidence, propose a specific change, compare fairly and carry the result forward.

We also learned that unsuccessful branches are useful records. They show which hypotheses failed and give the optimizer evidence to avoid blindly repeating an approach.

A useful evolution tree must explain both performance and execution state.

What's next for LoopBench

We want to expand into repository-level coding tasks, add more agent harness adapters and evaluate promising changes across repeated trials and held-out benchmarks.

We also plan to deploy the controller and persistent storage so campaigns can continue without a developer’s laptop. Today, inference and sandbox execution run in Nebius, while the demo controller runs locally.

Longer term, LoopBench should make it easy to export a proven harness version, reuse its lessons and trace every improvement back to the evidence that earned it.

Built With

• nextjs
• react
• sqlite
• typescript