Rewind-Bench: catching AI code fixes that only look correct

Same 30 bugs, two models. The real recorded data changed nothing on a small model and solved everything on a frontier one.
Every AI repair agent reports a pass rate, measured on the data it already saw. Rewind-Bench checks what survives data it never saw.
The agent left the wrong constant in place and added a second error that cancelled it out. The output matched. It was not fixed.
Three agents, same model and seed and source. The only variable is where their upstream data comes from — invented, or actually recorded.
One flipped comparison, three steps upstream, moves a loan-loss reserve by 21%. The final number is the only place you can see it.
The substrate: zaatool, a reactive notebook built months earlier. Cell state is already JSON, which is why recording it costs one write.
A notebook running as a live app. Rewind-Bench drives this through its public REST and MCP surfaces and never modifies it.
What anyone shipping a repair agent can use: hold out data, skip the agent loop on small files, and match evidence to model strength.
The bounds, stated rather than buried — including the four times the bench itself was wrong before it was right.

Inspiration

I built zaatool months ago because I wanted automation I could actually read — a reactive notebook instead of a canvas full of boxes and arrows. It was never meant to be research.

Then I went through the literature on AI agents that repair code, and the same difficulty kept appearing: recovering what data was actually flowing when the code broke. Everyone works around it, because on a normal notebook you have to snapshot a live Python kernel and diff its object graph, which is lossy and slow.

In zaatool, every value crossing a cell boundary is already JSON. Recording the exact state a cell saw costs one write. The tool I happened to own turned out to be the right instrument for a question other people can only answer expensively.

And once I could record that state, an uncomfortable question followed: does handing it to an agent actually help — and would I be able to tell?

What it does

Rewind-Bench injects one known bug into a notebook, hands it to three agents that differ in exactly one variable, and grades what comes back.

agent	may run code?	where its upstream data comes from
A · monolithic	no	reads the source and reasons
B · stepwise	yes	invents it, or pastes it in by hand
C · rewind	yes	the real recorded scope, injected

Then it does the part almost nobody does. Every repair that passes is re-run against a held-out seed the agent never saw. A fix that only matched the data it was shown is recorded as a lucky pass, not a success.

297 episodes: 30 designed bugs × 3 arms × 2 models, plus 39 bugs found in real notebooks. Every transcript is in the repo.

How we built it

TypeScript on Node 24. acorn for AST mutation, vitest for 120 unit tests that need no credentials. Inference through Featherless (OpenAI-compatible): DeepSeek-V4-Flash and GLM-5.2.

zaatool is the substrate and is not modified — Rewind-Bench drives it only through its public REST and MCP surfaces. Bugs are never written into real notebooks; every run works on a throwaway duplicate deleted in a finally.

Ground truth comes from mutation. An AST operator flips one operator, constant or comparison, and the mutant only enters the corpus if it still parses and genuinely changes the output. A quarter of candidates change nothing and are discarded.

Pipeline decisions that mattered:

Token accounting split three ways — prompt / reasoning / answer. Reasoning turned out to be 65–100% of completion tokens; a single total_tokens figure would have hidden the entire cost story.
Failures split into protocol / length / repair. Collapsing "the model produced malformed JSON", "the model got truncated" and "the model was wrong" into one bucket throws away data.
Fenced JSON instead of native tool calling. It costs us a protocol-failure category and buys portability — we swapped models with one environment variable and zero code changes.
Session tokens refresh on a mid-run 401; transport errors retry once and are counted, never folded into results.

Before any LLM was involved we measured whether replay is meaningful at all: given identical source and identical incoming scope, $r = 0.8942$ of cells return a bit-identical result (93/104 cells, 1,040 replays), with all 11 exceptions classified by cause.

Challenges we ran into

The instrument was wrong four times, and looked fine every time.

Each milestone was specified as a contract, implemented by a separate coding agent, then verified by re-deriving every number from the raw JSONL. That verification — not the implementation — was the work:

A corpus where 76% of bugs named themselves in the symptom (expected a1, got a1_v2). Nothing to diagnose; every arm scored 100%.
An instrument with no resolving power: two-thirds of mutations sat in cells that read nothing, so the "recorded evidence" handed to arm C was literally {}.
A difficulty axis pointing the wrong way. We banded by producer-to-consumer distance; in a linear chain that puts the "hardest" band on the last cell — exactly where the symptom already points. Fixing it deleted our best-looking result.
A lucky-pass metric so tautological it never fired once in 45 runs.

Three separate times an agent's written summary disagreed with the file it described. results/metrics.json is the record; prose is not.

The hardest part was staying honest when the first full run went against the hypothesis. It is recorded as-is — and it turned out to be half the finding.

Accomplishments that we're proud of

The held-out check caught our own favoured arm. The injected bug changed a penalty multiplier from 22 to 23. The agent never touched it:

// injected bug
let penalty = a.missedPayments * 23;

// what the agent submitted as its repair
let penalty = a.missedPayments * 23;                  // left alone
let bonus = Math.min(a.creditHistoryYears * 2.5
                     + a.missedPayments, 25);         // cancelled out

It added a second error of exactly the same magnitude pointing the other way. The output matched perfectly — until Math.min clamps at 25, which it does on the held-out seed. The other two arms failed this bug outright, so the only arm that "solved" it, hadn't.

A metric that only punishes your competitors is not worth having. This one punished us.

The bench also found a real concurrency bug in zaatool itself — a run could report success before its last cell result was readable. Diagnosed, reported upstream, mitigated client-side, and it resurfaced under --concurrency=3 exactly as predicted.

What we learned

1. Recorded evidence only converts if the model can use it. Identical 30 bugs, identical prompts and seed, only the model changed:

repairs that survived	small model	frontier model
A · monolithic	26 / 30	28 / 30
B · stepwise	27 / 30	27 / 30
C · rewind	26 / 30	30 / 30

Under the small model, arm C lost as many paired comparisons as it won. Under the frontier model it lost none, to either arm. Neither difference reaches significance at $n = 30$ (McNemar exact, $p = 0.25$ and $0.50$) and we do not claim it does — what changed is the consistency of direction, not the p-value.

2. It is still not free. Arm C cost 3.7× the tokens of an arm that never executed a line, to gain two repairs in thirty. If a file fits in the prompt, reading it beats every interactive strategy we tried.

3. Some bugs have no symptom at all. We audited every mutation for whether it changes the output an agent can actually see. 5 of 30 designed and 11 of 39 found mutations do not. On bugs that were observable, all three arms score 89–93% on the real-world corpus rather than the 69–74% the raw totals suggest.

What's next for Rewind-Bench

The obvious gap is size. The largest notebook here is 21 cells; everything fits in a prompt, which is exactly the regime where an agent that reads everything should win — and it does. The interesting question is what happens when the program no longer fits, because that is where recorded evidence should start paying for itself. No notebook that large existed on this instance, so we did not test it, and we say so rather than implying otherwise.

After that: native tool calling to remove the protocol-failure category, and a larger corpus so the direction we observed can be tested for significance rather than only for consistency.