Anneal

Landing
Agent Creation
Console
Agents
Inspiration

Every team we know builds agents the same way: write a prompt, try it on three examples, ship it, and hope. When you tweak something and it looks better on the four cases you happened to check, you have
no idea whether it actually improved or whether you got lucky. There is no scoreboard. We wanted to find out what happens if you give that process one.

## What it does

Anneal takes a job described in plain English and builds an agent to do it, then proves the agent got better.

You answer five questions in a browser: name it, say what it should do, pick what it can use, say what makes an answer right, and give a handful of examples. Anneal writes the goal file, the tool list,
the scorer and the task split itself. Then it runs a six-step loop:

- **Design** wires the same goal and tools into several different agents: one alone, a planner and a doer, a doer with a reviewer.
- **Try it** runs every design over your examples and scores each one, task by task.
- **Find the faults** reads the failures and groups them by what went wrong, naming the repair each fault calls for.
- **Fix one** applies that repair, never spending the same repair on the same fault twice.
- **Prove it helped** re-runs the challenger and the incumbent on tasks held back from every earlier step, and throws the change away if it does not win there.
- **Make it cheaper** walks the winner down to cheaper models for as long as the score holds.

Tools come from any Model Context Protocol server or your own Python functions, and the picker asks what the agent should be able to do rather than for a launch command.

## How we built it

Twenty-nine Agent Orchestrator sessions, each on its own branch, each taking one piece: the architect, the runner, the diagnosis, the repair operators, the gate, the annealer, the console, the onboarding
interview. Around 214 commits, 641 tests, and roughly 13,000 lines of Python. One of those sessions was not ours: Anneal spawned it itself to write a tool one of its agents was missing.

The stack is Python with FastAPI for the console, the OpenAI SDK pointed at whichever provider a model ladder names, scipy for the paired statistical test, and Ollama for running the whole thing locally
with no key at all. The demo film is Remotion, narrated by Sarvam.

Two rules held the whole build together. Nothing outside the gate may read the held-back split. And no number in the README, on the site, or in the deck is typed by hand: they are all generated from the
committed runs, with a test that fails the build if the two ever disagree.

## Challenges we ran into

**The gate kept telling us things we did not want to hear.** Accuracy moved in both directions across our domains. The honest response was to report every refusal in the results table rather than quietly
showing only the runs that went well.

**A concurrency bug that looked like a model failure.** One domain's tools kept state in module globals while the runner executed tasks in a thread pool, so every in-flight task read whichever state was
touched last. The entire failure story was a race, not the model. It happened to us twice, in two different domains, months apart in build time.

**The paired test was nearly blind at first.** Testing on pass^3 booleans meant a task moving from 0/3 to 2/3 registered as no change at all and contributed no evidence. Switching the unit to pass counts
recovered real statistical power.

**Building it in a terminal made it a terminal product.** Creating an agent through the browser left it invisible: the index only listed agents that already had runs, there was no way to start one, and
the success page printed a laptop file path and a shell command. Walking the app as a first-time user found six things like this, the worst being that opening an agent mid-run showed a different agent's
numbers under the right heading.

**Small local models could not always do the job.** On the meeting-room booking agent every design scored zero, because a 3B model answered with a clarifying question instead of calling a tool.