Attest

The phone agent that refuses to guess. One disclosed call, every answer citing the exact words that support it, and a calibrated confidence or an explicit abstention.

Comment

The record lies

Roughly half of the listings in a health insurance provider directory are wrong. A federal audit found about 50 percent inaccurate. A congressional secret shopper study called the mental health providers listed as in network and reached a bookable appointment in 18 percent of attempts. Re-audited listings were still wrong 44.8 percent of the time months later.

You call the number on your insurer's list and reach a nail salon. Nobody finds out until somebody needs care.

The record exists. The record lies. The only instrument that establishes current truth is a phone call to the published line.

A verification tool that guesses is just another source of bad data

Attest places one disclosed outbound call through CALL-E and does one of two things. It reports exactly what it heard, quoting the words verbatim with character offsets into the transcript. Or it says plainly that it could not establish an answer.

It never does the third thing, which is what everything else does: produce a confident-sounding answer nobody can check.

The abstention is the product.

The call that proves it

A real Atlanta counseling practice consented in writing to one automated call and to public use of the transcript, and chose to remain anonymous. We placed the call. It reached voicemail.

Here is what the system did with nothing:

  • Office name confirmed: unknown
  • Accepting new patients: unknown
  • Accepts plan: unknown
  • Verdict: unverifiable, posterior 0.50, which is the prior untouched

It learned nothing, and it said so.

The detail worth pausing on: the calling platform reported 0.88 confidence, labelled high, that the call completed. Completing a call is not learning an answer. A system that confuses those two reports a confident nothing, and that temptation was sitting right there in the payload.

The guarantee, measured

Every number below regenerates from a clean checkout with one command, on 300 calibration and 300 held-out scenarios, disjoint folds, seed 20260725.
Empirical coverage at a 90 percent target 90.3 percent (Wilson 95 percent: 86.5 to 93.2)
Abstains 57.7 percent of the time
Correct when it does answer 96.9 percent
Wrong if forced to answer everything 12.3 percent

That last row is the whole argument. The silence is what buys the accuracy.

Ablations, same folds. Remove hedge detection and accuracy falls to 87.8 percent. Remove the dead-end guard and it falls to 89.8 percent. Remove conformal calibration and you get 89.1 percent with no coverage guarantee at all.

Class-conditional check. The marginal average hid a real gap: "no" answers were covered only 83.2 percent, under the 90 percent target, while "unknown" sat at 100. Mondrian per-class thresholds lift "no" to 88.5 percent, overall 93.0. We ship the finding rather than averaging it away.

It survives contact with a real telephone

Simulation is not evidence. So we ran 36 pre-registered scripted calls over a real phone line to a consented builder line, with ground truth known at dial time. Eight were excluded by a deviation protocol written before analysis, leaving 28 attributable.

At the harness-calibrated threshold, never fitted on this data: coverage 28 of 28 (Wilson lower bound 87.9 percent), accuracy when answering 100 percent, abstention 42.9 percent. Counting every excluded call as a miss gives a worst-case floor of 77.8 percent, and that floor ships too.

The real channel abstained less than the simulation did.

What is actually wired

CALL-E is imported and called at runtime through the official Python SDK, in one obvious module (backend/app/calle/client.py). Not referenced. Called.

The reliability layer is self-built and survives deleting every hosted AI API from the architecture: a graded hedge lexicon, span-grounded extraction with character offsets, direct Fellegi-Sunter reconciliation with documented priors and a match-weight waterfall, and a hand-rolled finite-sample-corrected split conformal core you can read in about twenty lines.

302 tests, strict mypy, ruff, gitleaks over full history in CI. Security posture that assumes a hostile reviewer: SSRF resolve-then-pin on every server-side fetch, a webhook receiver that verifies HMAC over raw bytes with a freshness window when a signing secret exists and treats the platform's current unsigned deliveries as untrusted hints that can only trigger an authenticated re-fetch, phone numbers redacted in every public response and test-enforced, fail-closed key gating, and Ed25519-signed attestations that anyone can verify with the committed public key.

A maintainer reviewed the skill five times, found real bugs, and merged it

We opened the required pull request early rather than at the deadline. A CALL-E maintainer reviewed it within eight hours and filed seven blockers. Every one was real, and two were in the abstention mechanism itself.

The gate could serve a polar answer when the calibrated prediction set said {unknown}, because it never checked that the singleton agreed with the answer. Our own harness could not have caught it: it derives the answer as the argmax of the class scores, and the argmax is unknown across the entire region where the set is {unknown}, so the two forms are provably identical there. A sweep over a million points found zero divergences under argmax and 110826 under an extracted answer, which is what the shipped skill uses.

Separately, a later answer could be credited to an earlier question, producing a confident yes, not abstaining, span-grounded to a sentence about a different question. A wrong answer carrying a citation is worse than no answer, because the citation is what invites belief.

Four more rounds followed, each one probing the fix from the round before at exactly its scope boundary. The last one is the clearest: our identity check used a comma to tell a front-desk greeting apart from a question, so "Example Family Medicine, right?" read as the practice confirming itself. A tag question carries a comma too, so the protection covered the attack. Identity is now bound to the speech act and only an unambiguous self-identification, or an affirmative answering an identity question the agent actually asked, establishes it.

Before shipping that fix we attacked it ourselves with an adversarial pass, which found seventeen more, including our own new rule failing on the maintainer's exact sentence with the question mark deleted. Punctuation is a transcription artifact and the respondent does not control it.

Every finding across all five rounds is fixed, one commit each, every regression test run against the old code first to confirm it fails rather than passing vacuously. No reported number above changed: the eval output regenerates byte-identical. The pull request was merged into CALL-E's own repository on August 7, so the skill can be read on their main branch rather than taken on our word.

The most useful finding was the least dramatic. A documented example claimed a 43.3 percent abstention rate where the command printed 60.0. For a project whose entire claim is that its numbers regenerate from the code, a hand-typed figure contradicting the program refutes that claim better than any critic could. Both worked examples are now diffed against real command output in CI, so drift fails on the commit that causes it.

What a judge can actually do

  • Try it live. The console serves real recorded runs with transcript, highlighted supporting spans, verdict stamp, and the match-weight arithmetic.
  • Have it call you. A judge-gated sandbox dials the judge's own number, with consent required, one call per number ever, a hard global cap, and a kill switch. Both rails are enforced in one serialized transaction after a second-model review found them raceable.
  • Run it with zero credentials. docker compose up --build brings up the whole stack.
  • Put it in your pocket. Attest Pocket is read-only against the same production API.
  • Check the math yourself. One command regenerates every number and figure in this writeup.
  • Read the skill in CALL-E's own repository. The Agent Skill was merged into CALLE-AI/awesome-phone-call-agents on August 7, after five maintainer review rounds. Those rounds found real defects in the abstention mechanism, including a gate that could answer while the calibrated prediction set said unknown, and an identity check that treated an echoed question as a confirmation. Every one is fixed, each with a regression test written to fail against the old code first, and the review thread is public.

What we are not claiming

Being straight about the edges, because a verification project that overstates itself is refuting its own thesis:

  • The platform exposes no recording URL. We verified this field by field on a real terminal payload and by grepping the entire SDK. Run audio therefore exists only where we captured our own end of a consented call, and it is labelled as such. We reported this to the CALL-E team.
  • Webhooks arrived late and unsigned, so the poller stays authoritative. During our integration window webhook_url was accepted and never delivered (confirmed with a tunnel capture over 20 minutes); the platform turned delivery on July 29, without signatures. We measured delivery live on our own receiver on August 5, and because an unsigned delivery proves nothing about its sender, the receiver treats it as a hint that at most triggers an authenticated re-fetch. Both findings went to the CALL-E team.
  • Disclosed calling cannot measure how disclosure changes answers. There is no undisclosed baseline, by design. We log the respondent's reaction and report it qualitatively rather than claiming a causal effect.
  • The consenting practice reached voicemail because her line always does. She told us that before we called, and we agreed to those terms. It is the honest case, not a lucky one.

Who this is for

Attest speaks to patients, because the person harmed by a wrong listing is the person the interface is written for.

The buyer is different, and the public record says who. The New York Attorney General's December 2023 report ran a secret shopper survey of 13 health plans; of nearly 400 calls, only 14 percent produced an in-network appointment. That report became enforcement, including a settlement where 100 percent of called providers listed as accepting new patients were unreachable or not accepting, and another for 2.5 million dollars plus restitution. Both impose periodic directory audits monitored for years.

Somebody has to place those calls, on a recurring basis, and produce a record that survives being challenged by a plan's counsel. That work is people with phones and spreadsheets today.

Built With

Share this project:

Updates