call-rehearsal

Inspiration

A phone call has real-world side effects, and so does whatever the result triggers. Building an earlier CALL-E workflow surfaced a question that reviewing the script never answered: what does the automation do when the call does not end the way the happy path assumes?

The answer was worse than expected. A field the call never established is absent from the result, and an absent field is falsy once it reaches your code. A rule as ordinary as confirmed != false is therefore true when voicemail picked up and nothing at all was extracted. The courier gets dispatched because nobody answered the phone.

The same hole swallows a refused consent, a receptionist who would not pass the call on, and someone who is not the customer saying "yes, that's fine".

What it does

A call plan is the three things that together decide whether a phone workflow is safe to automate: the task the agent speaks, the result_schema the call returns, and the decision rule the surrounding automation applies to that result. Each is usually reviewed on its own. The expensive bugs live in the seam between them.

call-rehearsal projects a plan onto twelve realistic endings of a call and reports what the automation does on each one.

    Verified human confirmed               ->  dispatch the courier  (side effect)
    Verified human declined                ->  hold the order  (no side effect)
 !! Consent refused                        ->  dispatch the courier  (side effect)
 !! Wrong person answered and agreed       ->  dispatch the courier  (side effect)
 !! Gatekeeper would not pass the call on  ->  dispatch the courier  (side effect)
 !! Voicemail answered                     ->  dispatch the courier  (side effect)
 !! Nobody answered                        ->  dispatch the courier  (side effect)

  CRITICAL [voicemail] 'dispatch the courier' runs when voicemail answered
      The result {} (nothing was extracted) still resolves the decision rule to
      'dispatch the courier', which changes the real world.

Ten of the twelve dispatch the courier. Two of them are a real confirmation and a real refusal.

Findings are graded. unsafe-side-effect is critical: a branch that changes the real world runs on an ending that is not a verified, consenting yes. indistinguishable-from-confirmation is high: a non-confirmation produces byte-identical results to a real one, so no later audit can separate them. unrecordable-outcome means the schema has no field for reachability, identity, consent or deferral, so the distinction is lost before the result reaches the automation.

The exit code gates a workflow. 0 means the plan may go out, 20 means it should not, 30 means the plan could not be read.

How we built it

Standard library Python, no dependencies, no credentials, no network access.

Twelve outcomes are declared as frozen dataclasses over four semantic axes: whether a person was reached, whether identity was verified, whether consent was given, and what was agreed. Exactly one combination counts as a confirmation, and every branch marked side_effect: true that fires on anything else is reported.

The decision rule is evaluated, never executed. A call plan arriving in a pull request is untrusted input, so the expression is parsed to an AST and rejected unless every node sits on a small allow-list. No attribute access, no indexing, no arbitrary calls; the only callables are is_missing and is_present. A field the call never established resolves to a MISSING sentinel that is falsy and equal to nothing, which is how result.get("confirmed") already behaves in real automation code. Modelling that faithfully is what makes the skipped confirmation visible.

Field roles are declared rather than inferred, following the repository principle that a workflow must not guess critical values. Guessing the decision field wrong would rehearse the wrong thing and then report a clean run, which is worse than no rehearsal at all. --suggest-fields offers candidates for a person to choose between and selects none of them.

29 tests, all offline.

Challenges we ran into

Deciding what not to build. An earlier design simulated the conversation itself, which needs a model in the loop and ends up testing the simulator more than the plan. Reasoning about the shape of the result instead keeps the tool deterministic and reproducible, which is what makes it usable as a gate on every commit.

The second was scope honesty. This says nothing about whether the agent's phrasing works on a live line. calle-script-advisor covers the task text and voice-preflight covers what the critical lines sound like when spoken. call-rehearsal covers what happens to the result afterwards, and the README says so plainly rather than implying wider coverage.

Accomplishments that we're proud of

The demo finds a believable and expensive bug in seconds without dialling anyone: a delivery-confirmation plan that ships the order when voicemail picks up. The fix is two changes, and the same tool proves the fix landed.

Running it needs no CALL-E credit, no phone number, and no supported locale, so anyone can reproduce the result.

What we learned

Absent is not no. Automation that treats a missing field as a negative answer is usually safe. Automation that treats it as anything else usually is not, and neither reading is visible from the schema alone. Giving reachability, identity and consent their own fields is what makes the difference auditable months later, when somebody asks whether the customer really agreed.

What's next for call-rehearsal

An explicit termination outcome returned by CALL-E on every call would remove the need for most of this analysis, and that suggestion went into the hackathon feedback form. Until then the outcome library is the portable version of it.

The next build step is platform adapters, so a plan can be rehearsed straight from an n8n or Dify node rather than only from a file.

Built With

• call-e
• python