Rehearsal

Rehearsal lets an AI agent plan a batch job in a simulated copy of the company. A person approves that exact plan once, and a model-free executor applies it, holding any step that went stale.

Inspiration

Agents that write to billing and support systems fail in ways nobody reviews until the job is already done. Approving every call stops a person dozens of times and never shows them the whole job. Cleaning up afterwards cannot unsend a ticket reply, and restoring a snapshot can wipe a teammate's legitimate edit.

Teams already feel this. In Georgian's Wave 3 benchmark, 55% of technical decision makers reported an AI incident in the last 12 months. 75% use human approval gates, and only 29% use policy-as-code. The threat they name first is excessive agency.

We wanted the opposite shape. The agent may decide. It decides inside a copy of the company. A person approves the whole plan once, and production only ever receives that approved plan, applied by an engine that has no model in it.

What it does

A billing bug double-charges a set of customers. Rehearsal gives finance and support one place to fix it.

The agent investigates Stripe and Zendesk in a planning world and proposes the whole remedy: refunds, ticket updates, private notes, and a summary. Every amount is exact. Every step says whether it can be undone, how the write is protected, and which read it depends on. The plan is grouped by customer, and it is bound to a canonical hash of that exact content.

A reviewer works on the plan, not on a stream of tool calls. Unchecking a customer drops the note that depended on that customer's refund. Refunds over the policy threshold need a finance lead. Where the policy asks for two seats, two different people sign. The approval is stored against the hash, so a changed plan needs a new approval.

Apply is a separate act. A deterministic executor, holding no model key, sends the approved steps to a second world that started from the same company data. Before each step it re-reads what that step relied on. If a charge was edited, or a new matching charge appeared after approval, that step is held and so is anything that depended on it. The rest of the plan goes through. Every step ends in a receipt: the plan hash, the approver, the outcome, and the reason.

The same loop covers the cases that make a batch job hard:

Records created in the rehearsal get new ids in the operational world, and later steps use the operational id.
A throttled refund is retried with the same request and lands once.
A response that never comes back is reconciled by reading, not by sending the refund again.
If a later step fails, the engine restores only a field that still holds the value it wrote. A teammate's edit in between is left in place and recorded as a conflict.
A second remedy for the same obligation is refused.
A reviewer can promote a decision into a precedent, such as "this account gets a credit, not a refund." Later plans cite it. If that precedent changes, the old approval no longer covers the step.
The approved plan can also be run in a disposable world first, so the team can read the report before anything is sent.

Sign-in is Clerk. Authorization is ours. The app asks a separate identity service, over mutual TLS, whether this person may invite, approve, apply, or promote. Owner, finance lead, support lead, and member each hold a fixed set of permissions. One person belongs to one organization. The header shows who they are. The buttons follow what they are allowed to do, and the server checks again on every command.

How we built it

The product is four tracks that meet at one plan.

Rehearse. Two Era worlds, planning and operational, are seeded with the same incident and reset to the same instant. Stripe, Zendesk, and Slack are write-enabled simulators with tenant isolation. The planner is a bounded tool loop. Its tools are typed: refund, ticket update, note, summary. The model cannot invent a URL, a method, or its own evidence. Each read is stored with what came back and the query behind it. Each write is stored with the before and after. Those records become the plan, the dependency graph, and the hash.

Review. A React app, behind a Fastify API, renders the plan by customer. Policy runs on the diff: thresholds, dual approval, and which steps are irreversible. Clerk identifies the person and their organization role. Approving, revoking, inviting, and changing a role are API commands. The grant stores the plan hash and the permissions that were held at approval time.

Apply. Postgres keeps the plan, the grants, and an append-only journal. The exact request bytes and the idempotency key are stored before anything is sent. pg-boss carries the apply job to an executor worker whose environment has the operational world and nothing else: no model key, no admin token, no Clerk secret. The worker walks the graph in order, admits each step, dispatches, and writes a receipt. Admission re-checks the grant, the hash, the world, and any cited precedent.

Guard. Before a step is sent, the engine re-runs the observation and the query. A new row that matches the original query is a phantom, and the step is held. Compensation walks backward only through fields this run still owns. Injection scripts plant the phantom charge, the teammate edit, and the throttling, so the same story can be shown on demand.

Identity sits beside this, not inside the model. The IAM service loads a versioned rule file. The API and the executor call it with their own client certificates. The planner never receives a certificate or the IAM address. Unknown rules, missing organizations, and transport failures deny. They never fall open.

Contracts between these pieces are Zod types. The hash is RFC 8785 canonical JSON. Money is integer cents. The web app is Vite, React, and Tailwind. Tests run on Vitest, including real Postgres and pinned Era images.

Challenges we ran into

The interesting problems were the ones that look small and decide whether the receipt is true.

The rehearsal and the operational world do not share ids. A refund created in the copy is a different Stripe object from the refund created later. The note still has to name the operational refund. Steps refer to earlier results symbolically, and the executor fills in the id that production actually returned.

A lost response is not a failed send. The refund may have happened. Retrying it would apply the remedy twice. The journal already holds the exact request, so a restart reads the world, finds the effect, and confirms the step once.

The world moves between approval and apply. An edited ticket is the easy case. A charge that did not exist at approval time, and now matches the same query, is the one a hash cannot see. Re-running the query at admission catches it, and the dependent note is held with it.

Undo has to know what it owns. Restoring a ticket field is wrong if a teammate changed that field after we wrote it. The engine re-reads, reverts only its own value, and records the conflict when the value is no longer ours.

Deciding and doing had to be different processes. The planner's environment can reach a model and the planning world. The executor's environment can reach the operational world and IAM. Each process starts from its own env file, and startup refuses the other side's secrets.

Who may click is not the same as who Clerk says they are. Organization role is identity. Permission to approve a finance step, apply a plan, or invite a member is a rule we version and evaluate in IAM. The grant records the permissions that were true at approval, and apply checks those permissions again.

Accomplishments that we're proud of

The whole loop is real. An agent planned a billing remedy in the rehearsal world, a reviewer approved it in the app, and the executor applied every step.

On the same twenty-eight tasks, against pinned Era images, Rehearsal confirmed 27. Direct writes confirmed 14. Modelled per-call approval confirmed 19. Rehearsal recorded no invalid allows and no duplicates, and it did that with 51 approvals against 627 modelled per-call prompts.

The moments we built the product for all run on those simulators: a phantom charge held after approval, a teammate's edit preserved through compensation, a throttled refund applied once, and a lost response reconciled instead of repeated.

Around that loop we shipped the review app, hash-bound grants, two-person approval where policy asks for it, precedents a finance lead can promote, execution rehearsal in a disposable world, and an IAM service that decides permissions for the API and the executor over mutual TLS. One person, one organization. The server is the gate. The screen only reflects it.

What we learned

An approval is a fact about one exact plan, not a mood the executor inherits. Bind it to a hash, store who held which permission, and make the applier re-read the world before it sends anything.

The useful unit of review is the job, not the call. A person can see eight customers, drop one, and watch the dependent note disappear. They cannot do that across dozens of yes/no prompts.

A step is only as safe as the query that selected it. Re-checking the row you already saw is not enough. You have to ask the query again, because the dangerous record is the one that showed up later.

And the model belongs on one side of the door. Once the plan is approved, production should be a replay with receipts, not another conversation.

What's next for Rehearsal

The next plans should arrive already shaped by the precedents this team has promoted, so a finance lead's decision on Monday is cited, checked, and visible on Friday's diff.

The same approved plan should keep opening as an audit: the policy it was signed under, the hash, the two people, and the receipt for every step, including the ones that were held.

From there, more of the company's systems join the same graph, and operators run the loop on their own incidents: rehearse, approve once, apply, and read the evidence.

Clerk

Clerk is the identity provider. It is not the authorization engine.

A person signs in with a Clerk session. The browser sends that session token as a bearer on every API call. Clerk also holds the organization and one organization role: owner, finance lead, support lead, or member. Owner is the role that creates the organization. Member is the default when someone is invited. Clerk's own permissions stay the built-in ones, and only the owner can manage membership. We do not create Rehearsal permissions inside Clerk. An unmapped Clerk role, including the old admin role, has no Rehearsal permissions at all.

Authorization lives in a separate IAM service that speaks Clerk. The API and the executor call it over mutual TLS, each with its own client certificate. The browser and the planner have no certificate, so they cannot ask. The Clerk secret stays inside IAM. The app never tells IAM who the person is. It sends the session token, and IAM verifies that token with Clerk, reads the user, the active organization, and the role, and maps that role through a versioned rule file onto permissions such as rehearsal:approve:finance_step, rehearsal:apply:plan, and rehearsal:invite:member. The caller cannot send the rule text.

Every person-facing command asks IAM first. Inviting a member and changing a role go through our API: IAM allows it, and only then does the API call Clerk. The organization is the one on the session. A plan that belongs to another organization is not found. Where policy asks for two approvers, the two seats are permissions held by two different people, not a single Clerk role. The grant stores the permissions that were true at approval time and the rule version, so a later edit to the rule file does not rewrite an old signature. Apply checks those stored grants again, with no person and no second Clerk call, before anything is sent.

If IAM is down, the token is bad, the rule is unknown, or the answer does not echo the question, the action is refused. The screen shows the role Clerk verified and hides what that person may not do. The server is the decision.

Era

Era is the company the agent rehearses in, and the company the approved plan is applied to. They are two write-enabled worlds, seeded with the same incident and reset to the same instant. Stripe, Zendesk, and Slack are the systems in both.

The planner only ever reaches the planning world. The executor only ever reaches the operational world. The admin token that can reset, inject, or fault a world stays on the API. Each worker proves itself with its own secret, and the API mints a short-lived tenant token for that worker's world only. Nothing falls back to a shared tenant name. A planner that somehow held an operational URL or the admin token refuses to start. An executor that held a model key refuses to start.

Inside a rehearsal, every read comes back from Era and is stored with the query that produced it. Every write is stored with the before and after. That record is the plan. After approval, the executor sends the same steps to the operational world. Ids differ, so a note written later names the refund Era created in production, not the one from the rehearsal. Between the two moments we can change the operational world on purpose: a new matching charge, a teammate's edit, a throttle. The executor re-reads Era before it sends, holds what no longer matches, retries what Era asked it to wait on, and reconciles a response Era accepted but the process never saw. The scorecard that compared direct writes, per-call approval, and Rehearsal ran on these pinned images.

SurrealDB

SurrealDB Agent Memory is where a reviewer's decision becomes something the next plan can recall.

A finance lead promotes a precedent from a completed run. One example is "accounts in collections get a credit, not a refund." The statement, the conditions, and who approved it are written as knowledge on a concept named for that precedent, in a memory scope that belongs to one organization. Publishing writes the canonical record, reads it back, and refuses an older version or the same version with different content. Recall returns the matching precedents with the fact ids and a trace id, so a later plan can show which memory a step is using and a reviewer can see it before they approve.

The hosted service passed that gate: publish and readback, recall with a trace, replacing a version while the history keeps the old one, a neighbouring scope that cannot read or change ours, and a plan that cited the old version held once the new one was approved.

Postgres keeps the event log. A precedent is not active until a sync job has published it to Agent Memory and read it back. The write key for that job stays with the API. The executor holds no memory credentials. At apply time it does not ask recall whether a step is allowed. It checks the citation against the approved registry, so a superseded, withdrawn, or changed precedent holds the step, and so does a new precedent in the same area that recall never surfaced. Memory is the index a plan can cite. The approval is still the authority.