Self-Improving Agent Loop

Inspiration

We kept seeing the same thing in agent demos. Someone tweaks a prompt, the score goes from 60 to 65, and they call it a 5% improvement. But LLMs are random. Run the same agent twice on the same tasks and you get different scores without changing anything. So that 5% might be real, or it might be the same coin landing differently. Almost nobody checks. We wanted to build the thing that checks.

What it does

You give it a goal, a set of tools, and a scoring function. It writes an agent, runs it against held-out tasks, looks only at the failures, works out the main reason they failed, changes one thing to fix that reason, re-measures, and keeps or reverts based on the result. Before any of that, it runs the baseline three times without changing anything to measure how much the score moves on its own. That becomes the bar. Any gain smaller than it gets rejected as luck. On extraction that bar was 0.0067. A change gained 0.0238, three and a half times the bar, and was kept. The next change lost 0.0119 and was reverted.

How we built it

Five stages in a loop. Synthesize writes an agent spec. Evaluate runs it and records every tool call, token count, and elapsed second. Diagnose reads only the failures and assigns each one a cause from a closed set of seven, with no "other" bucket. Mutate makes a single change aimed at whichever cause dominates. Select keeps or reverts against the measured noise floor. Python and pydantic. Four domains: code and math checked by executing against hidden tests, structured extraction from messy documents, multi-step API chains where later calls need earlier results, and a real MCP server whose tool schemas we discover
at runtime rather than hardcode. The engine imports none of the domains. They arrive through two functions, get_suite() and get_evaluator(). We froze the four data shapes before writing any stage code. That decision paid off within minutes of starting parallel work, when two components had independently defined the same interface incompatibly. Caught early it cost one message.

Challenges we ran into

One domain scored exactly 0.0000 on every run. It looked like a hard benchmark. But it cost 234 tokens a run, the cheapest of three, on the domain needing the most steps. A multi-step task cannot be the cheapest unless it isn't taking the steps. The runner had _NoTools() hardcoded and the agent was never handed anything to call. Once wired properly it scores 0.7222. Our keep threshold defaulted to 1e-9, which accepts any positive number. So the system would happily accept a gain of 0.0625
while its own reliability measurement said the score wanders by 0.1179 on its own. We moved the threshold derivation into the loop so it comes from measured variance instead of a constant.
We also had to reseed with a deliberately weak baseline. Starting near the ceiling leaves the loop nothing to find, and a
correctly working improver will correctly find nothing.

Accomplishments that we're proud of

The system caught four bugs in our own project, and our tests passed the whole time. The best one: we audited every change it had made, twelve of them, and seven gave a reason its own trajectory data contradicted. One justified doubling the step budget by claiming runs were being cut off at 8 steps. The agent had stopped on step one. It never ran out of anything. We also caught a Goodhart effect. One accepted change improved the score it optimizes while a different score fell. We could tell drift from damage because we had measured both noise floors, not just one. And the diagnosis generalizes. On live runs it identified output_format_violation for extraction and premature_stop for the other two domains, from run structure alone, without knowing which domain it was looking at.

What we learned

Measurement and testing are different things. Our test suite was green through every one of those four bugs. Numbers that didn't add up caught what tests couldn't. A strict gate that rejects almost everything is not the same as a broken improver, and you cannot tell them apart without evidence. Ours accepted a real gain and caught a real regression in consecutive generations, which is what proves it discriminates rather than just refuses.Negative results are worth keeping. Episodic memory adds 29.6 tokens a task and saves nothing on single-turn work, because there are no execution steps to prune. That tells us exactly where it should pay off instead.

What's next for Self-Improving Agent Loop

Production is the honest gap. Every evaluator we wrote needs a known right answer, and production never has one. The fix implicit signals like thumbs-down, users rephrasing the same question, and abandonment. The diagnosis stage carries over unchanged because it reads run shape rather than correctness. Reliability does not, because you cannot replay a live user three times.Then finish the memory experiment. We predicted memory only pays off where it prunes tool calls and never got to test the one multi-step domain. That arm is sitting in the repo honestly marked "not measured". Bigger changes after that. Right now it rewrites prompts and swaps strategies. It cannot wrap a tool in argument validation or add a retry on schema failure. Structural changes move numbers; wording usually doesn't. And more replicates rather than more tasks. The noise floor shrinks with replicates, and right now that floor is what blocks everything else.