-
-
Second defect: place-call.ts reads response.result.end_reason, and 0 of 15 responses carry that field.
-
The whole run, with no API key, no network and no node_modules: 0.483s, printed by the shell.
-
The body of the 429 is the only place the API ever shows the remaining allowance: 20 a day, and a refusal still spends one.
-
Behaviour eight across the corpus: 6 of the 7 connected calls do it, and no other project ships a payload that contains it.
-
The captured call. The recipient's last two turns arrive after the agent has already said goodbye.
-
First defect: errand.ts matches on words the API never sends, so its voicemail branch is unreachable.
-
What the platform returned for that call: taskCompleted true, confidence 0.95, high.
Three minute walkthrough: https://youtu.be/tAwQ147nacg
The call that started it
I placed one call through the CALL-E API to see what it returned. The agent said "Thank you. Goodbye." and hung up. A second later the recipient said "Yes. I can hear you clearly."
The platform recorded that call as taskCompleted: true, confidence 0.95, label high, and summarised it as "The recipient confirmed they could hear the automated call clearly, so the recorded result is yes."
The confirmation arrived after the agent had gone. The verdict was right about the words and wrong about the conversation, and nothing in the response said so.
That is the shape of the whole problem. This API emits things a caller would not predict from its documented shape, and the only way to find out is to have seen one. So I started keeping them.
What it is
Conformance reads the JSON payloads a project already ships and reports which of the CALL-E API's real behaviours that project has never been tested against.
$ ls node_modules
ls: cannot access 'node_modules': No such file or directory
$ time node src/replay.ts ../../..
corpus: 15 real responses, 8 behaviours declared as predicates.
6 projects carry call-shaped payloads to score against them.
A dot means the behaviour never appears in that project's payloads.
project n 1 2 3 4 5 6 7 8
--------------------------------- -- -- -- -- -- -- -- -- --
apps/python/casechaser 7 . . . . . . x .
apps/python/redline 1 . . . . . . x .
apps/python/ringdown 2 . . . . x . x .
apps/typescript/calle-conformance 15 x x x x x x x x
plugins/zapier-calle 3 . . . . . . x .
skills/verify-by-phone 1 . . . . . . . .
53 projects call this API and ship no JSON payload for this checker to read.
real 0m0.483s
No install, no API key, no network. Node 22 executes the TypeScript directly, and the only dependency in package.json is used by the optional probes, not by the checker.
The eight behaviours live in src/quirks.ts as executable predicates rather than prose. One definition labels the corpus, verifies the rewriting of every fixture, and scores third-party code, so a behaviour cannot be documented without a function that decides it.
What it found
Two defects in code already merged into this repository, each with the file and the line.
apps/typescript/call-on-behalf/src/errand.ts:87. The failure handler branches on "voicemail", "machine", "answer", "busy" and "unreachable". Every failure code the corpus actually contains is a bare SIP number: 404, 486, 603. None of those strings ever matches, so every failure falls through to the default and errand.ts:125 is unreachable. A caller who reached an answering machine is never told they reached an answering machine.
apps/typescript/hirecall/src/lib/place-call.ts:258. The guard reads response.result?.end_reason to decide whether response.result can be trusted. The corpus contains calls that never connected and still carry a populated structuredResult with zero transcript turns, which is exactly the case the guard exists to catch. end_reason appears in none of the fifteen responses, so the app's own parser invents the field and defaults it to failed. The guard then asks whether it is no_answer. It never is, so the guard misses, and the value it trusted was never platform data at all. An earlier version of this entry called the check circular; that framing was withdrawn on 8 September 2026 and rewritten in docs/found-by-the-corpus.md.
Neither of these is a code smell. Both are a fixture away from being obvious, and until this corpus existed there was no fixture to be a fixture away from.
What I measured about the free tier
While building the corpus I ran out of allowance without ever reaching a person, so I measured why.
Every request that reaches the planner spends one unit of the 20-per-24-hour allowance, including the ones the planner refuses. No number is dialled, no phone rings, no call object is created, and the allowance drops by one anyway.
Two instruments, neither of which places a call. src/watch.ts sampled the counter while at the cap: eleven consecutive readings held at 20 of 20 across two and a half hours, and capacity returned per unit at +24h, which rules out a fixed daily reset. src/headroom.ts then issued requests that reach the planner and are refused there; two were accepted and the third met the limiter. Both rest on payload validation running before the rate limiter while destination screening runs after it.
Nothing in the API exposes the remaining allowance. There is no usage endpoint: /v1/account, /v1/balance, /v1/credits, /v1/usage and /v1/me all return 404. The number appears in exactly one place, the body of the 429 that tells you it is gone.
This is one account, one region, one free tier, over one window. It is a measurement, not a documented guarantee, and the README states the three conditions that would falsify it. npm run headroom re-runs the second instrument in about a minute.
How I built it
The checker is TypeScript with no runtime dependency, run directly by Node 22. Fifteen real API responses were captured, then rewritten for publication by src/mask.ts, which refuses to emit a fixture whose behaviour set changed under rewriting. fixtures/README.md is generated from the manifest, so the documentation cannot drift from the predicates. There are 54 tests across four suites, including one that feeds hostile input and asserts the tool never reports success.
The API key is treated as a destination rather than a setting. src/endpoint.ts validates it: https only, and only a CALL-E origin. Anything else is refused with a message saying why. Twenty-six tests cover that one file, including a path that tries to smuggle a different destination past an allowed origin.
The film is Remotion, so the video is React and every frame is code. The matrix that prints in the terminal is the same object that becomes the logo at the end: a WebGL tile field via @remotion/three that reorders itself into a five by five grid with one cell missing. The mark is not a metaphor for the tool, it is a frame of the tool's own output.
The captured call is reconstructed as audio, because the API returns no recording of a call. The transcript is real and word for word; the voices are synthesised, and the film says so on screen rather than letting a viewer assume otherwise.
What I learned
A finding is only worth what its artefact is worth. The first draft of this asserted a runtime of 0.447 seconds because it was in my notes. I measured it: 1.536s cold, 0.483s warm. Every figure in the film and the README now traces to a file someone else can open, and the ones that could not be traced were cut.
The interesting behaviour of an API is the part its documentation has no reason to mention. A rate limiter that counts refusals is not a bug and not a secret. It is simply not the sort of thing a shape description describes, and it costs you a day of work the first time you hit it.
53 of the 59 projects here that call this API record nothing at all. A project with no payload is not missing six behaviours out of eight. It is uncovered by all of them, and no dot in a table can say so, so the report says it in a sentence underneath.
Challenges
Getting the evidence to survive contact with the truth was most of the work.
Building a corpus of real responses on a tier that gives you twenty requests a day, when refused requests also cost one, meant every capture had to be planned before it was spent.
The eighth behaviour, the one the whole film is built on, only appears if you look at when turns arrived rather than at what they said. Six of the seven connected calls in the corpus do it. It took having seven of them side by side to see it at all, which is the argument for the corpus in one sentence.
And a small one I am keeping as a lesson. One shot in the film highlighted end_reason on a line that actually uses endReason, the derived variable. The scene printed a token that was not in the source, inside a film whose entire point is that nothing on screen is fabricated. I found it, fixed it, and it is the reason every value in every frame is now read from the checkout instead of retyped from memory.
Log in or sign up for Devpost to join the conversation.