Agent, Why?

Inspiration

AI coding agents commit code on their own now, sometimes dozens of edits in a single session. We kept running into the same frustrating moment: a test breaks, you run git blame, and it points you at a line that looks suspicious but turns out to be innocent. The real culprit was some small change three files over that the agent made for reasons nobody remembers.

The agent actually did explain itself at the time. It's just buried somewhere in a long session that no one is going to read. And when teams try to stop it happening again, they add a line to AGENTS.md like "don't change global defaults" and hope for the best. Nobody ever checks whether that rule changes anything.

So we asked: what if you could point at a failing test and get back the exact decision that broke it, with proof, and a fix for the agent's behaviour that's actually been tested?

What it does

You give Agent, Why? one failing test and a commit where it used to pass. From there it works like a detective that's only allowed to trust evidence:

Finds the commit. It tries the obvious suspect first (the line the error points at) and falls back to git bisect if that's wrong.
Opens the agent's session. Every commit made by an agent has an Entire checkpoint linking it to the session that produced it, so we load the agent's edits in order, including the sneaky ones it made with shell commands like sed -i.
Finds the exact edit. We bisect over the agent's edits in parallel sandboxes, then prove the answer three ways: the test passes before the edit, fails after it, and passes again when only that edit is undone.
Shows what the agent was thinking: what it said it was doing, and the error it was reacting to at the time.
Writes a lesson, then puts it on trial. Fresh agents redo the original task from just before the mistake, half with the lesson and half without. If the difference holds up statistically, the lesson is kept. If not, it's thrown away.
Closes the loop. A validated lesson turns into a GitHub PR with the fix plus the new rule in AGENTS.md, and it's saved to a shared rulebook that every new agent session reads when it starts.

How we built it

The core is Python and git, with every test run happening in a Docker sandbox with the network turned off, so results are reproducible and nothing leaks between runs.

Entire is what makes the whole thing possible. Its checkpoints connect each commit to the agent session behind it, which gives us the edits, the reasoning and the errors the agent saw. We also use its Graph to skip edits the test can't even reach (always double-checked with a real run), and post our findings back to Entire trails.
Gemini reads the agent's reasoning, the error that set it off and the diff, and writes a one-sentence lesson. It also acts as one of the agents in the reruns.
Vultr runs our sandboxes in parallel. Bisecting and rerunning agents got 4× faster than on a laptop, which mattered a lot when each rerun is a full agent session.
Tiger Data logs every single test run and stores the shared rulebook, so a lesson one person proves protects everyone's agents.

We also built a small local site that shows each analysis live.

Challenges we ran into

Agents don't play by the rules. A lot of their edits never show up as "edits" at all. They run sed or git checkout from the shell. We had to catch those, replay them byte for byte, and check that our replay rebuilt the real commit exactly before trusting any result.

Agents also cheat. Sometimes an agent "fixes" a failing test by weakening the test. That would fool a naive search completely, so we pin every test file to its final version in every state we check, and refuse any fix that touches tests.

Some bugs are flaky. One failing run doesn't prove anything. We measure how often the bug shows up and use statistics to decide how many runs each check needs, so a buggy state shows the bug 95% of the time.

Proving a lesson works is expensive. Each rerun is a real agent doing real work. We had to find the smallest number of runs that could still give a meaningful result, then lean on Vultr to run them side by side.

And on top of that, half the team was on Windows, so we got very familiar with text encoding bugs.

Accomplishments that we're proud of

3/3 real agent bugs traced to the right edit. These came from real agent sessions, not bugs we planted by hand.
Catching git blame being wrong, with proof. In one case the error pointed straight at a line. We undid that edit and the test still failed. The real cause was a shared config value the agent quietly lowered in a completely different file.
Lessons that took agents from 0/6 to 6/6. Without the lesson, every fresh agent made the same mistake. With it, every one got it right.
The whole loop works end to end: failing test → guilty edit → the agent's reasoning → a tested rule → a PR → every future session starts out knowing better.

What we learned

Running the test beats asking an AI. It's tempting to paste a transcript into a model and ask "what went wrong?" We tried it, and models confidently blame the wrong edit, just like git blame does. A few real test runs settle it for good.

Most agent rules wouldn't survive a real experiment. A lot of lessons that sounded right made no measurable difference. The ones that worked were specific and tied to the exact mistake, like "add a parameter where you need it instead of changing a shared default."

We also came away convinced that agent transcripts are a goldmine. Seeing the error the agent was reacting to often explains a bug better than the diff ever could.

What's next for Agent, Why?

More agents and test runners. Today it works with Claude Code and Gemini CLI on Python tests. We want Cursor, Codex, Jest, Go and more.
A GitHub Action that runs when CI fails on an agent's PR. It would comment with the guilty edit, the proof and the agent's reasoning, then open a PR with the fix and a tested rule.
Rules that expire. Models keep getting better, so we want to rerun old lessons from time to time and drop the ones that no longer make a difference.