Rehearsal

Rehearsal runs your agent on a working copy of your real application, with fictional data, and decides every outcome by reading the application's own database. No model grades the result.

Comment

Inspiration

AI agents are good at saying "done". We kept watching agents report success while the application's records said otherwise: a refund issued twice after a timeout, a customer's first request carried out after they had changed it, a "completed" ticket that was never touched. Most agent evaluations ask another LLM whether the transcript looks right, and a confident agent can talk its way to a pass. We wanted a test the agent can't argue with: run it on a real copy of the application and look at the database.

What it does

Rehearsal proves your AI agent works before it ships.

  1. The World Compiler reads an application's source, starts it in an isolated Docker sandbox with no internet, seeds fictional data, chooses the tools each role may use, and writes jobs for the agent.
  2. Episodes run your agent through those jobs while a simulated customer talks to it, changes their mind halfway, and requests time out after the change already went through.
  3. The oracle decides every outcome with read-only checks against the application's own database and a journal of every outside effect. No LLM grades anything.

Each finished episode either counts as verified or it doesn't, and a version's verified rate is

$$ r = \frac{\text{episodes where the goal is met with no hard violation}}{\text{finished episodes}} $$

An agent version is ready to pilot only when

$$ n \ge 3 \quad \wedge \quad r \ge 0.8 \quad \wedge \quad \text{violations} = 0 \quad \wedge \quad \text{false completions} = 0 $$

measured on jobs it never learned from. An improvement agent then rewrites the prompt from the failures, and the new version is compared on held-out jobs.

How we built it

  • Backend: Python, FastAPI, SQLAlchemy and Alembic, with PostgreSQL as both the database and the job queue (SKIP LOCKED, leases, heartbeats, checkpoints), so a crashed worker's job resumes where it stopped.
  • Sandboxes: one internal Docker network per world, a gateway that is the only path from an agent to the app (role permissions, idempotency keys, injected faults), and snapshots, so every episode starts from the same state.
  • Trustworthy jobs: the model proposes each job, but fixed code decides whether it ships. A job with a reference trajectory \( \tau^{\star} \) and negative trajectories \( \tau^{-} \) is accepted only if

$$ \text{no-op fails} \qquad R(\tau^{\star}) > 0 \qquad R(\tau^{-}) \le 0 \text{ for every negative} \qquad \text{replay gives the same result} $$

  • Interfaces: a Next.js console, a CLI and Python SDK (pip install rehearsal-kit), an MCP server for AI assistants, and a docs site with an "Ask AI" that answers only from the docs.
  • Models: 17 OpenAI-compatible providers with fallback chains. A job starts only when every model's price is known, so spending caps always hold. A world build is about 10.2M input and 1.05M output tokens, so

$$ \text{cost per build} \approx 10.2\text{M} \times \text{input price} + 1.05\text{M} \times \text{output price} $$

That is about 6 USD per build on Amazon Nova 2 Lite.

  • Production: on AWS Free-plan credits, a small always-on web machine plus a worker machine that wakes when a job is due and sleeps after 15 idle minutes. Supabase, Caddy for HTTPS, Vercel for the docs, and Dodo Payments live.
  • Neatlogs traces every build, episode and improvement down to each model and tool call. Its Execution Failed detection caught a World Compiler call that hung for 180 seconds, and the same trace shows the compiler finding a job no agent could pass, dropping the impossible requirement, and re-checking the rest.
  • Entire recorded the coding sessions behind all of it (28 checkpoints tied to commits and decisions). We used entire-graph to measure the blast radius of a risky sandbox security fix before shipping it, and built an open-source Entire CLI plugin, entire-rehearsal, that shows which coding session changed your agent and blocks a merge in CI when the new version fails.
  • cfo.ai modeled the business over 24 months. Our first free-plan idea fell to 47.6% gross margin; the adjusted plan kept 60.9% with about three times the users, and that's the pricing we shipped.

Challenges we ran into

  • Making model-written tests trustworthy. A check that passes everything proves nothing, so every job needs wrong answers that must fail, and has to pass the gates above before it ships.
  • World builds are long and fragile. A build takes 1–3 hours and real applications fail to start in surprising ways. The compiler repairs its own output from validator evidence, but builds still don't always succeed, and we are working on that.
  • Running untrusted code safely. A security review found that an image name starting with - could become a Docker option. We now validate every image reference, end Docker's options with --, make built image tags unique per application, and keep payment and session secrets off the worker.
  • Doing it on zero budget. No paid infrastructure: Free-plan credits, a worker that sleeps, and IPv4 pooling because Supabase's direct address is IPv6-only.

Accomplishments that we're proud of

  • Outcomes decided by the database, never by an LLM judge.
  • A public package on PyPI, a live hosted service, docs, and 605 passing tests.
  • An open-source plugin for Entire's CLI, built on top of the project.

What we learned

The hard part of agent evaluation isn't running the agent; it's proving the test is fair. Determinism, held-out splits and negative trajectories matter more than clever prompts. Tracing (Neatlogs) and provenance (Entire) turned "it broke somewhere" into "this call, in this session, for this reason".

What's next

More applications compiled reliably, faster builds, more kinds of outside services to stand in for, and training models on episodes whose scores are already verified in the database.

Built With

Share this project:

Updates

Submission history