confidence-gated-results

Inspiration

A phone call can succeed, bill normally, and hand back nothing usable.

The call connects. The agent talks. The status comes back completed. And the extraction produced null — the customer mumbled, the line was noisy, the question was ambiguous, or the model simply guessed and returned a confident-looking value that is wrong.

Every integration we looked at treats status == completed as "I have the data." That is the bug. completed describes the call. It says nothing about whether the result is trustworthy enough to act on.

What it does

confidence-gated-results sits between a CALL-E result and whatever your system does next, and forces an explicit answer to a question most code never asks: is this good enough to act on?

It classifies every result into one of four outcomes:

The key distinction — and the one that took the longest to get right — is REPAIR vs REJECT. A call that reached a human and produced nothing usable is not the same as a call that never reached anyone. The first deserves a follow-up question; the second deserves an error. Collapsing them into one "failed" bucket is how systems end up re-dialling real people at 9pm for no reason.

How we built it

A pure function with no I/O, no clock, and no network, wrapped in a thin adapter for the CALL-E result shape.

• classify-result.mjs — the whole decision surface. Deterministic: same input, same output, always.
• Schema-aware completeness — if you passed a result_schema to CALL-E, the skill knows which fields were required and can tell "partially extracted" from "extracted nothing," rather than treating any missing key as total failure.
• Confidence thresholds — per-field and overall, configurable, with defaults that fail toward REVIEW instead of toward ACT.
• 19/19 tests, no network, no credentials.

The bias is deliberate and one-directional: when the gate is uncertain about its own certainty, it escalates to a human rather than acting. An unnecessary REVIEW costs someone thirty seconds. An unwarranted ACT ships the wrong order, cancels the right appointment, or bills the wrong customer.

Challenges we ran into

Confidence scores are not uniformly available. They are not always present, and not always meaningful when they are. The skill treats a missing score as unknown, not as zero and not as fine — unknown routes to REVIEW. Assuming either extreme produces a system that is quietly wrong in one direction.

Ambiguity is not failure. The first version collapsed "no clear answer" into REJECT, which is how you end up re-dialling a customer who already answered — just not in a way the parser liked. Splitting REPAIR out from REJECT was the single most valuable change in the design.

Async status is not result status. An HTTP 200 from call creation is a queue acknowledgement. Even a terminal completed is a statement about the call's lifecycle, not about the payload. The skill only ever inspects the payload.

Accomplishments that we're proud of

• 19/19 tests green, fully deterministic, no network required.
• The four-way outcome is genuinely novel in this repo — every other integration we read is a two-way success/failure branch.
• It composes: the companion entry (plugins/shopify-flow-cod-confirm, PR #62) hit exactly this problem in a concrete domain, which is what motivated generalising it.
• scripts/validate_repository.py passes: "Repository validation passed."

What we learned

The interesting failure mode of voice agents is not the call that fails. Those are loud and you handle them on day one. It is the call that succeeds and returns garbage — because that path looks identical to the happy path right up until the wrong thing gets shipped.

Making a system state its own uncertainty, explicitly and in a typed way, turns out to be most of the engineering.

What's next for confidence-gated-results

Per-field repair prompts (re-ask only the field that failed, not the whole script), calibration against observed human-override rates, and a small adapter so the four outcomes map cleanly onto common workflow engines.

Links

• Pull request: https://github.com/CALLE-AI/awesome-phone-call-agents/pull/63
• Demo video: https://youtu.be/vcdPnS7qvwg
• Companion entry: https://github.com/CALLE-AI/awesome-phone-call-agents/pull/62

Built With

• call-e
• confidence-scoring
• human-in-the-loop
• javascript
• node.js
• rest-api
• voice-ai