landing
logo

Inspiration

Every team building an AI agent hits the same wall. The agent is great in the demo, then the first real customer breaks it. The customer changes their mind halfway through, and the agent finishes the old request. A request times out after it already went through, and the agent does it twice. Or the agent says "done" when nothing changed.

Most agent evaluations ask another model whether the transcript looks right, and a confident agent can talk its way through that. We wanted the test a careful engineer would run by hand: put the agent inside a working copy of the real application, give it real jobs, make things go wrong on purpose, and then check the database to see what actually happened.

What it does

Rehearsal is CI for AI agents. It works with any web application that has a repository and container images: a help desk, a task manager, a CMS, a team chat, an ERP, a store.

World Compiler: point it at your application. It reads the code, boots the app in an isolated sandbox, seeds it with fictional data, picks the tools each role may use, and writes jobs it proves can be both passed and failed.
Grounded in real use: before writing a single job, it searches the web for how real people use that application, so the jobs look like real requests, rules and pitfalls.
Rehearsals: your agent works through the jobs while a simulated customer talks to it, changes its mind and resends requests. Failures are injected on purpose, like a rate limit or a timeout after the change already went through.
Verdicts from the database: every episode ends with read-only checks on the application's own records. Was the goal reached, exactly once, and does what the agent claimed match the data.
Why it failed, in plain words: each failed episode gets named causes such as "acted on the old request", "did the same thing twice" or "said it was done when it was not", with the usual fix.
Fix my agent: a loop writes the next version of the agent from its failures, keeps it only if it verifies more episodes without breaking a rule, and proves the best version on held-out jobs no round has seen.
Agent Factory: no agent yet? Rehearsal writes one for your application, hardens it with the fix loop, proves it, and hands you a runnable agent.py with a PROOF.md of every score.
Pull request checks: rehearsal gate and a GitHub Action rehearse a changed agent next to the version on main and fail the check on any regression.
Incidents become tests: describe a mistake an agent made in production and Rehearsal writes a validated job that catches it in every later run.
Bring any agent: test it by its OpenAI-compatible URL, drive episodes from your own code in any language, or start from a ready-made world without building anything. Python and TypeScript SDKs, a CLI, and an MCP server for Claude Code and Codex.

How we built it

The control plane is FastAPI with PostgreSQL, which doubles as the job queue. Workers pick up builds, evaluations and fix loops, and run each application inside Docker sandboxes. The console is a Next.js app with a live 3D view of every build and episode (three.js), so you can watch each role at work. The client ships as rehearsal-kit on both PyPI and npm.

The core rule is that no model grades an outcome. A job is published only after it passes three gates: its reference run passes, its known-wrong runs fail, and a live agent can pass it. An agent's score is then a plain count:

$$ \text{verified success} = \frac{\text{episodes whose database checks all pass}}{\text{episodes run}} $$

The fix loop keeps a new version $ v_{k+1} $ only if it verifies more episodes than $ v_k $ on the training and dev jobs without a new rule violation. The final comparison happens on held-out jobs that no round has seen.

NVIDIA Nemotron: the right model for every role

Rehearsal has many roles that need very different things, so each role gets its own Nemotron model and its own reasoning setting.

Nemotron 3 Ultra, reasoning on, sits where a wrong answer wastes a whole build: designing jobs, judging whether a job is fair, repairing broken builds, and writing the next version of an agent in the fix loop.
Nemotron 3 Super, reasoning on, does the heavy tool work: reading the source, working out how to run the app, seeding data, writing tools and checks, and playing the agent under test.
Nemotron 3 Nano, reasoning off, plays the simulated customer. The customer speaks on every turn of every episode and only needs a sentence or two. With reasoning off a turn takes about 0.5 s and 28 output tokens, instead of 2.7 s and 191 tokens.

Every role has a fallback chain inside the Nemotron family, so one model having a bad moment does not stop a build. Tool calling works on all three sizes with the same schemas.

Nebius Token Factory: one API, every call measured

All inference runs on Nebius Token Factory through its OpenAI-compatible API. We run no GPUs ourselves.

Switching every role to Nemotron was one provider entry: a base URL, a key and the model names. No prompt or tool schema had to change.
Ultra, Super and Nano share one key and one tool-calling format, so moving a role to a bigger or smaller model is a one-line change.
The per-request reasoning switch lets the same model family run fast for the customer and carefully for judging.
Clear per-token prices mean every build, evaluation and fix loop gets a spend cap before its first call, and the console shows what each build step and each role cost.

Tavily: grounded in how people really use the app

Tavily, now part of Nebius, is called at runtime in three places of every build.

Research before jobs: a researcher role reads docs, forums and issue trackers about the application through the Tavily Search and Extract APIs and writes a brief of real requests, rules and pitfalls, each with its source. The job planner must base at least half the jobs on that brief, and every job links the pages it came from.
Setup notes: before planning how to run the app, the compiler searches the application's own documentation site for how to run it in Docker, so first-run settings stop showing up as failed starts.
Errors that keep coming back: when the same application error repeats during a build step, the compiler looks it up the way a person would paste it into a search engine, and gives what it finds to the model's next try.

Pages are read once at build time and frozen into the world version, so episodes never touch the live web and results stay comparable.

Challenges we ran into

Real applications are messy to start. Many need an install lock, a config file or a first-run step buried in their docs before the API answers. We added setup notes from each project's own documentation, early checks for host settings and unfinished installs, and clearer messages when a fix lands on the wrong service.
Agents that get stuck burn money. In long builds a model would sometimes retry the same failing step again and again. We built a no-progress brake that pauses the step and explains why it is stuck, spend shown per step, and a web lookup for errors that keep repeating.
Malformed JSON. Long structured answers sometimes came back with a missing bracket or extra text. A tolerant parser plus a repair pass fixed it.
Restarts during builds. Deploying new code used to kill running builds. Workers now hand their job back on shutdown, and the build resumes from its checkpoint without losing an attempt.
Safety with user URLs. Testing an agent by URL means our workers call an address a user gave us, so we refuse private and internal addresses and store the agent's key encrypted.

Accomplishments that we're proud of

A full practice world compiled end to end through all ten phases, with validated jobs, its own tools and roles, and more than half its jobs grounded in real forum and documentation pages found through Tavily.
A catalog of real open-source applications to compile, from a help desk and a task manager to an ERP and a store, plus one application held out that the compiler is never developed against.
A real rehearsal where the customer left out a detail, the upload hit a rate limit on purpose, the customer sent the request again, and the agent checked before acting instead of doing it twice. The database confirmed it.
A CI gate that failed a pull request at 67% verified success against a required 80%, with the named causes in its report.
A production-style incident, an agent closing the original issue instead of the duplicate, turned into a validated test in three tries.
One product, many doors: the console, the CLI, the Python and TypeScript SDKs and the MCP server all do the same things.

What we learned

Checking a model's work with code beats asking another model to grade it. Exact error messages fed back to the model fix most problems in a few tries.
Routing by role matters. The smallest model with reasoning off is the right choice for a chatty customer, and the biggest model is worth it only where a mistake costs a whole build.
Grounding jobs in real user requests changes what gets tested. The pages Tavily found pointed at limits and pitfalls we would never have guessed from the source code.
Long agent sessions need brakes. Without them, a stuck step can quietly spend a budget.

What's next for Rehearsal

More ready-made worlds for popular open-source apps, so teams can rehearse an agent without building anything.
Faster builds by reusing work across world versions.
Running each build as its own isolated job once serverless jobs support containers.
Deeper CI: comments on the pull request with the episodes that regressed, linked to their replays.
A public benchmark of agents acting inside real applications, built from Rehearsal's worlds.