HOLDLINE

HOLDLINE calls a list of places, asks them all the same thing, and gives you back only the answers the call actually established. The rest come back empty, with the reason.

Comment

Inspiration

I found it in CALL-E's own issue tracker.

Issue #316: a developer asked their agent to confirm an address. The agent never raised it. The result came back populated anyway — address_correct: "unclear", valid against the schema, high confidence. They only spotted it because the summary happened to mention the agent moving on.

That stuck with me. A call result tells you what the model concluded. It doesn't tell you what was said out loud, and there's nothing in the response that separates asked and not understood from never asked.

I'm not the only one here who noticed. verify-by-phone grounds each answer in a transcript span, verity-verification-core gates a task_completed claim behind an independent read-back, incident-escalation-call only records what the recipient actually said. Checking the transcript is becoming the house style, and it should be.

What's still missing is the distinction underneath that check. Not confirmed and invented are different states, and a gate that treats them as one is unusable — mine accused 100% of paraphrased questions of being fabricated on its first run. So I built the thing that checks, and then I measured the thing that checks.

What it does

A hospital discharge coordinator has a patient who's medically ready to leave and can't, because nobody has confirmed a bed. So they ring eight care homes, one at a time, through eight phone menus and eight hold queues, asking the same two questions.

HOLDLINE asks all eight at once, and then does the part that matters: it checks every answer against the transcript the agent actually spoke.

Verdict Meaning
verified The agent raised it and a usable answer came back, quoted.
asked_but_unclear Asked, no usable answer.
unattributed Something was asked and answered, but no probe for this field matches it.
attributed Nothing matched, but one question and one field were left over, so it can only be this.
never_asked Nothing was asked that this value could answer. A value here is flagged and withheld.
no_transcript Nothing to check against.

A withheld field comes back null, with the reason. A missing answer is a gap you can see. A wrong one is invisible until a patient spends another night in a hospital bed.

How I built it

TypeScript throughout: the CALL-E SDK, an MCP server, and a small console with no framework and no build step.

The decision that shaped everything else was the transport. The SDK takes an injectable fetch, so I implemented the wire contract — POST /v1/calls, GET /v1/calls/{id}, snake_case bodies, the Idempotency-Key header — as a fake and handed it to the real CalleClient. The engine talks to the genuine SDK. Only the network underneath is swapped. The whole suite runs with no credentials and places no calls, and anyone can clone it and see what I see.

I made the fake adversarial rather than convenient. It reproduces CALL-E's documented failure modes, each traced to a public issue: a confident result for a question that got skipped (#316), a call that stays queued past the client's patience and dials afterwards (#283), one that never reaches a terminal state (#305), a long silence before the agent speaks (#295), and a replayed idempotency key that answers 201 Created with the existing call (#315).

That's where the parts I trust most came from — especially sorting failures into retryable, deterministic and reconcile, because a timeout doesn't mean no call happened, and retrying one rings a second real person.

Challenges I ran into

My own measurement told me the core feature was broken. I wrote an evaluation harness over 400 labelled cases to turn "the gate catches invented values" into a number. First run: it caught 100% of them, and accused 100% of paraphrased questions of being fabricated. Topic matching is lexical, and an LLM agent paraphrases constantly. In production it would have called nearly every call a lie.

The fix was a new verdict, unattributed: count the question-and-answer exchanges no probe claims, and only flag a value that has no exchange left to have come from. Not confirmed and invented are different things and I'd been treating them as one. I wouldn't have found that by thinking about it harder.

The task field is capped at 255 characters and I couldn't find it documented anywhere. That's tight once a task has to carry a goal, a menu-routing hint and an AI-disclosure line. I compile the task from prioritised segments and drop the lowest first — and fail loudly rather than truncate a required instruction and ship a call that asks half a question.

Regions are not uniform. Not every country is callable, and not every callable country supports every language. I'm in Argentina, which isn't on the list — I found that out by trying. The supported-regions table is transcribed into the engine now, so an unreachable number gets refused locally with the list of what would work instead.

Then I placed 29 real calls, and six of them broke something I had written down as true. This is the part I'd most want read.

The first came back verified on a value that was a whole sentence saying nothing had been established. My usable-value check only knew short tokens like unknown and n/a, so a refusal written as prose sailed through. The unit suite missed it. So did 400 evaluated cases, because every value in that corpus was one word long.

The second was a phone tree reading my own outbound caller id back to me. The transcript recorded it in national notation, my redaction only matched E.164, and a full number reached a file the tool calls shareable — under a claim that every number in every response is masked. One notation is not every notation.

The third caught the gate crediting a field nobody had asked about. The agent opened by naming the very thing it had been sent to find out, never asked anything, and those words happened to sit inside the probe. A statement of purpose isn't a question, and I'd been counting it as one. That one's worse than the other two: it credited a field on less evidence than the gate claims to require.

The fourth took the architecture away. I built the engine on what looked like the obvious primitive — CALL-E takes one task and a list of recipients, so a fan-out is one dispatch, one idempotency key. Every test passed. It worked for a week.

Then I dialled three real numbers in one dispatch. All three connected and ran, and came back task_completed: true at confidence 1.0 with a populated result each. Every one of them returned zero transcript turns. The spoken content existed only as a prose summary.

The gate did the one thing it's for: it refused all three. Reading the summary instead would have been exactly what I accuse everyone else of. Correct, and useless — a batch that can verify nothing isn't a product.

So I dialled the same three numbers again, one call per target:

Same three numbers, same account One call, three recipients Three calls, one each
First run 0, 0, 0 7, 15, 7
Second run, both arms back to back 0, 0, 0 8, 14, 6

It reproduces, and npm run probe:fanout runs both arms so anyone can check. I rebuilt the dispatch around it: one call per target, sent together, each deriving its own key from the same authorizing record. That turned out better than what it replaced — reconciling one target can no longer re-dial the other two.

My fake had been hiding it the whole time. I built it from the documentation, so it returned transcripts for every recipient of a multi-recipient call. A fake built from documentation models the documentation. Only a telephone models the telephone.

The fifth was my own pipeline disagreeing with itself. Four more calls, two lines, and every field came back never_asked — correctly, because the agent never put a question of its own on any of them. On one it worked a menu and gave up. On another it answered the other party's questions for seven minutes without asking one.

The gate credits a field only when a turn actually asked for it. The compiled task — the instruction my engine writes for the agent — carried a goal, a routing hint and a disclosure clause, and nothing that told the agent to ask. The probe phrases did reach the API, but only as schema descriptions, which guide extraction and never reach the conversation.

So I'd built a judge that demands evidence and a compiler that never requests it. No test caught it, because my fake returns transcripts already full of questions. No labelled case caught it, because every case starts from a transcript that exists. A pipeline can disagree with itself in the one seam no fixture covers: the instruction that leaves the process, and the judgement that comes back.

The sixth was the same mistake from the other side. Partway through a hold queue the agent finally asked — in plain interrogative order, and the turn arrived cut off before the question mark. The gate called it never asked and threw the field away.

Worse is what it did count. Three other turns from that same call were logged as questions, because each carried a mark: the agent asking whether the transfer had failed, asking for a repeat, asking whether it could be heard. Questions about the call. The one question about the subject was the one it lost.

That defect sits in direct tension with the third one, and the tension is the fix. That one says a statement of purpose naming the topic must not count. This one says an inverted clause about the topic must. The difference isn't the vocabulary — it's the word order, and only at the start of a clause.

I fixed all six in the same order every time: add the case to the corpus, measure the damage, then change the check. The last one showed the gate eating 44 of 44 genuine questions and accuracy sitting at 78.0%. After the fix: 0 of 44 eaten, statements of purpose still caught 44/44, nothing never-established credited, accuracy 89.0%, and real answers withheld cut from 88 to 44. The suite pins both directions, so the second number can't be bought with the first.

And then, on the last day, one where all of it worked. CALL-E published an inbound testing hotline that morning — their own line, answered by a conversational agent, offered to entrants for exactly this. It's the first line I called that could hold up its end: a recording can't be asked anything, and a customer-service tree hands you to a queue.

One call, thirty-two seconds, two fields. The agent asked both out loud. The gate found a supporting turn for each and quoted it back, then split them: the line's purpose came back verified, because the question was put and a usable answer followed. Whether the call was being recorded came back asked_but_unclear and was withheld, because the question was put and the line answered that it didn't have that information.

That's the whole thing on a real telephone, and the first time every part of it worked at once outside the simulator. Every earlier live call was missing a piece. It also settles the fix from the same day — the clause telling the agent to ask out loud is what produced those two quotable turns, and without it both fields would have come back never_asked, like the four before them.

Accomplishments I'm proud of

I published my own failure rate.

What a caller ends up believing:
  Trusting structured_result   355 answers, 178 never established   50.1% wrong
  Through the gate             133 answers,   0 never established    0.0% wrong
  Real answers withheld         44

  Invented values caught          45/45   100.0%
  Direct asks passed              45/45   100.0%
  Paraphrases wrongly accused      0/44     0.0%
  Prose non-answers caught        44/44   100.0%
  Mentions without a question     44/44   100.0%
  Questions without a mark eaten   0/44     0.0%

Half of what a plain schema check would hand the caller was never established by the conversation. The gate returns none of it. It also withholds 44 real answers it can't attribute, and that line prints on every run. A benchmark that only contains the cases you handle isn't a benchmark.

And it works on a telephone, not only against my own fake. On CALL-E's testing hotline the gate verified one field with the sentence that established it and withheld the other with the reason, in thirty-two seconds. Everything above is measured offline; that one was measured on a phone line I didn't control.

167 tests, no network, no credentials. Three production dependencies. And the skill is merged: it lives upstream at skills/holdline in CALL-E's own awesome-phone-call-agents, through three rounds of Must Fix from that repository's maintainer — one of which made me reframe the skill because it claimed commands it couldn't run, and one of which made me rewrite the public history twice to get every live-call artifact out of it.

What I learned

CALL-E's webhooks aren't signed. The SDK says so itself, in a deprecation note on its own signature helpers. So my receiver treats a delivery as a doorbell rather than a document: it reads one call id, refuses any id it didn't dispatch, and asks the API what actually happened.

A timeout is the dangerous case, not the boring one. Most clients sort failures into retry or give up. On a telephone you need a third bucket that means go find out what happened.

Not confirmed and invented need different words. Half of what I built is that distinction in one form or another.

Some behaviour isn't in the docs until it's in your bill. I told the agent not to accept a transfer to a person, twice, worded two different ways. It accepted both times and waited on hold until the platform cut the call. That turned out to be by design — CALL-E waits when a call is transferred to a human — which is reasonable and is also written down nowhere. There's no maximum-duration parameter either, so a workflow that must stay inside an automated system isn't something you can build and then bound. I found that out by arithmetic.

A test suite built from documentation tests the documentation. Six times a real phone call found something 167 tests and 400 labelled cases could not. That's the lesson I'd keep if I had to throw the rest away.

What's next

Pressing keys. A real carrier line stopped accepting speech, required a keypad selection, and hung up on an agent that only had a voice. I watched it happen twice more on two unrelated systems, with the agent saying the digit out loud instead of sending it — once announcing that it was pressing. HOLDLINE doesn't press keys, and that's written down as a limit rather than dressed up as a feature, but it's the clearest thing 29 live calls taught me about where this breaks.

Then a Slack action so a ward clerk can ask from where they already work, durable stores behind the ledger interfaces, and better probe tooling to push that withheld-answer number down.

Built With

Share this project:

Updates