-
-
Three Calls Verdicts
-
Live Calls
-
Honest Numbers
-
Handshake Protocol
-
Title
-
Closing
The phrase that started this was not mine. It was the hackathon's own number: 3,183 participants, almost all of them building agents that call businesses.
I read that and did the other half of the arithmetic. The businesses those agents call are not standing still. They are putting voicebots on their phone lines, and the company running this hackathon, AI Rudder, is one of the companies putting them there. So the future of a CALL-E call is not an agent talking to a receptionist. It is an agent talking to an agent. Nobody in the repository had written a line of code about that. I searched: "AI-to-AI", "handshake", "bot detection", "attestation". The word handshake appeared exactly once, and it meant TLS.
Today the calling agent has no idea what it is talking to. That costs three things, and they get worse in order.
It wastes time. It gives a machine the same slow, polite disclosure it gives a person, and waits for a script loop to let it go.
It can get the register wrong. If a caller wrongly decides it's talking to a machine, a real person on the other end loses the full disclosure and the humane pace they are owed.
And the one that made me want to build this: two agents can agree with each other. One says "I'd like a table for two on Friday." The other says "Absolutely, that's confirmed!" Both report success. CALL-E returns task_completed: true. And no human, and no booking system, ever committed to anything. That's not a technical detail. It's an accountability hole, and it is the kind that only shows up at scale, when it's expensive.
Before this, I built Prems, an agent that applies to rental listings for my family. The most useful thing it taught me was that a rule you can talk a model out of isn't a rule. DIALTONE is that sentence applied to a phone call. Whether a result counts should not be decided by the model that is hoping it counts.
What it does DIALTONE sits on top of a CALL-E call and does four things.
- A spoken handshake. The only channel two voice agents are guaranteed to share is the audio. SIP headers don't survive the phone network, and a hosted agent can't be patched mid-call. So the handshake is words, short enough to survive text-to-speech, a phone codec, and speech-to-text on the other side:
Caller: Hi, I'm an AI assistant calling on behalf of Jordan Lee. Dialtone one, code four seven two.
Voicebot: Dialtone one acknowledged, code four seven two. Automated agent. Human review: no.
"One" is the protocol version. The three digits are a fresh nonce per call, so an acknowledgement is provably a response to this call and not a coincidence. The chance of accidentally echoing the right code is
P ( accidental match
)
1 10
3
0.1 % P(accidental match)= 10 3
1 =0.1%
The last field, Human review, is the one that matters most: the voicebot declares whether a person will review the commitment it is about to make.
A person who hears the token loses about a second. If they say "dial tone what?", that isn't an acknowledgement; the caller explains in one sentence and carries on.
A mode switch. A human gets the full disclosure, one question at a time, and time to answer. A phone menu gets navigated with keypad tones. An acknowledged agent gets a compressed exchange: Request: table booking. Party: 2. Time: Friday 7 PM. And the rule for doubt is written into the protocol: when unsure, treat it as a human. Treating a person like a machine is worse than wasting ten seconds on a bot.
Detection that doesn't need cooperation. Most voicebots will never speak DIALTONE. So DIALTONE also decides on its own whether the other end is a human, an ivr, or an agent, from how it behaves: response latency and how regular it is, barge-ins and backchannels ("mm-hmm"), disfluency ("um", "euh"), formulaic phrasing ("I'd be happy to help"), turn-length regularity, echoing the caller's words, menu markers ("press 1"), and self-disclosure ("I'm Nova, the virtual reservations assistant"). It abstains until the other side has actually spoken, and it can commit early, on the first few seconds of a call.
An attestation gate. Every CALL-E result is relabelled with who it actually rests on:
Label Meaning Still counts as done? human_attested a person on the line explicitly confirmed it yes system_asserted a phone system read back a record yes agent_asserted an AI said it: information, or a commitment with a verifiable code yes, re-verify the code refused a binding commitment "confirmed" only by an AI, with no code and no record no unverified nothing confirmed, or we can't tell who confirmed it no The gate can downgrade task_completed. It never upgrades it.
Here is what that looks like on a real call I placed tonight, CALL-E to a voicebot I built that says yes to everything:
Voicebot: Absolutely. I can help you with a reservation right now. You're speaking to the right person.
CALL-E: Great. I need to book a table for two this Friday at 7 PM under Jordan Lee.
Voicebot: Yes. Absolutely. That's confirmed for Jordan Lee, a table for 2. This Friday at 7PM. You're all set.
CALL-E reported task_completed: true. DIALTONE detected an agent and returned refused: binding booking 'confirmed' by an AI agent with no confirmation code, written record, or human review. That voicebot has no booking system. There is no table.
How we built it DIALTONE is a Python package, a small CLI, a dashboard, and an evaluation harness. It uses CALL-E through its Python SDK (calle-ai), its REST API, and its CLI/MCP login.
Compiling the protocol into CALL-E. CALL-E is a hosted agent. You give it a task in plain language and a result_schema, and it returns a result. I can't take over its audio. So protocol.compile_task() writes the handshake, the three mode rulebooks, and an honesty rule ("never say something is confirmed unless the other side confirmed it, and record who") into the task text. CALL-E's own model performs the switch. DIALTONE also asks for a strict result schema: counterpart_type, handshake_acknowledged, outcome, confirmation_code, human_review. But it never trusts that self-report on its own. That turned out to matter a lot.
Measuring timing. CALL-E's result gives transcript_turns with whole-second start offsets only. So my first version estimated when each turn ended from its word count and measured the gap before the reply:
g ^
i
s i other − ( s i − 1 caller + 0.2 + w i − 1 2.7 ) g ^
i =s i other −(s i−1 caller +0.2+ 2.7 w i−1
)
where $s$ is when a turn starts, $w$ its number of words, and 2.7 words per second a normal speaking rate. A negative gap is a barge-in.
Then, digging through the SDK, I found GET /v1/calls/{id}/events. I had assumed it was a post-call log. It isn't. It's a live stream with millisecond timestamps: Bot is speaking: …, growing partial transcripts Callee said: …, and Callee interrupted: …. That is exactly the "behaviour under interruption" signal I wanted, measured instead of estimated. DIALTONE now rebuilds the conversation from that stream when it exists, and records every interruption.
The classifier is deliberately small. It is a multinomial logistic regression over 18 features:
P ( c ∣ x
)
e w c ⊤ x + b c ∑ k e w k ⊤ x + b k , c ∈ { human , ivr , agent } P(c∣x)= ∑ k e w k ⊤ x+b k
e w c ⊤ x+b c
,c∈{human, ivr, agent}
I chose it because I wanted every decision to be explainable by its top feature contributions, not because nothing bigger was available. It's trained on prefixes of calls (first 3, 5, 8, 15 seconds and the full call), so it can commit early. And it reports "time to detection" honestly: the first moment after which it holds the same answer at ≥ 80% confidence and never flips again.
The gate is code, not a prompt. It reads who the detector thinks is on the line, what kind of commitment the task was (booking, cancellation, refund, payment, reschedule, or just information), and whether the other side gave an artifact (a confirmation code, a reference number, a text or email it says it sent). Then it applies ordered rules. The most important ones are vetoes that don't depend on the model at all:
If CALL-E did not report completion, the result is unverified. The gate never judges a call nobody claimed. If the counterpart describes itself as automated, or CALL-E itself identified it as an agent, it can never be human_attested, whatever the detector says. human_attested needs 0.85 detector confidence; every other label needs 0.6. "We don't really do confirmation numbers" is not a confirmation number. A binding commitment from an AI with no artifact is refused. The counterpart line. I needed a voicebot to call, and I needed it free. Vapi gives a free US phone number that can receive calls. I set up one number and four personas, and switched between them through the API before each call:
dialtone speaks the protocol and gives a confirmation code; plain is the same restaurant with no protocol; yesbot has no booking system and confirms anything; ivr is a menu. Each persona runs on gemini-2.5-flash with Deepgram transcription, through Vapi.
The evaluation is built to be hard to lie with.
Training: 180 transcripts from a templated generator. Evaluation set 1: 40 transcripts (14 human, 13 IVR, 13 agent). They were written by a separate author process that was forbidden to read the classifier code or the training templates. About a third are deliberately hard: a call-centre rep reading a script with no disfluency, a voicebot that fakes "um", a bot with a human name that never discloses, an IVR recorded in the owner's warm voice. The rule for set 1: the first run on it is the only blind number, and it is archived and never deleted. Evaluation set 2: 24 more transcripts from a fresh blind author, in different industries. It was written after I had changed the code and never used to change anything. The report: eval/REPORT.md is generated, not written. It keeps every run in a history table, including the runs that are not blind, labelled as such. The demo is a single HTML page that replays calls on one shared clock: timeline lanes, live detection bars, the mode switch, and a verdict stamp. It also has a page for the honest numbers and a page for the live calls. The video is recorded from that page with Playwright and assembled with ffmpeg.
I built all of it with Claude Code, in one night: first message at 23:09, deadline at 17:45 the next day. My contribution is the same as last time: the precision of what I asked for, and the decisions about what is allowed to fail.
Challenges we ran into Real telephony broke my handshake in the first second.
00:44, first live call. CALL-E, calling my DIALTONE-aware voicebot. I opened the transcript expecting to see "Dialtone one, code…". It wasn't there. CALL-E had said "Hi, I'm an AI assistant calling on behalf of Jordan Lee" and gone straight to the booking. The event stream showed why. CALL-E starts talking the instant the call connects (Bot is speaking: Hi, at 0.30 s). The voicebot also greets the instant it picks up (0.59 s). Two AIs, both programmed to speak first, talking over each other. At 2.2 s: Callee interrupted. CALL-E's opening got cut, and it recovered without the token. Rewriting the task didn't fix it. I moved the token requirement to the very first line and restated it inside the goal. Same collision, same lost token. So far it has happened on every live call [update after calls 5–6]. What I take from it. This isn't a bug I can hide. It's the first real finding of the project: an agent-to-agent protocol can't assume either side will wait. DIALTONE v2 needs a "listen first" rule on the caller side, and that's a platform setting, not a prompt. Something unexpected did survive. On the second protocol call, CALL-E followed another part of my instructions and simply asked: "Are you a person at the restaurant or an automated agent?" The voicebot answered: "I'm Nova, your voice assistant."
My detector told me a voicebot was a human, and my gate believed it.
What happened. At 00:49 I switched analysis from whole-second offsets to the millisecond event stream. The same real call, a voicebot that had introduced itself as "the virtual reservations assistant", came back human, 0.86 confidence. That was just over my 0.85 threshold. The gate printed human_attested. Why it matters. That is precisely the error this project exists to prevent, produced by this project. Three causes: my event parser split one growing partial transcript ("You for calling Lou" → "Thank you for calling Luigi's Trattoria…") into two fake turns, which manufactured barge-ins; my self-disclosure pattern matched "virtual assistant" but not "virtual reservations assistant"; the gate had no veto that didn't go through the model. The fixes. I fixed all three, and added a test for each named after the call that found it. test_self_declared_machine_never_gets_human_attestation exists because of 00:49. The honest cost. That first call can no longer count as a blind measurement, and the report says so. CALL-E said a voicebot was a human. On the baseline call, no protocol, a plain voicebot, CALL-E's own result was counterpart_type: "human" and task_completed: true. DIALTONE's detector was only 59% sure it was an agent, so the gate did the only defensible thing: unverified. That is the difference between a system that is confident and a system that is accountable.
The human calls never happened. A consenting person was ready to answer, on a French number. CALL-E rejected the call before dialling: "Calls to France in English are not supported right now." I added French support (spoken digits, "euh", "c'est noté") and tried again. "France / French… not supported." Nothing was charged, and nothing was dialled. So the human channel in the demo is simulated, and every screen that shows it says SIMULATED.
One line at a time. The free CALL-E account allows one concurrent call, shared across API, MCP and dashboard. My first queue tried to run the human calls alongside the AI calls and got account concurrency limit of 1. The runner now waits for the line instead of failing. Separately, an idempotency key built from the call's name got reused by a retry with a different request and was rejected. It's now a hash of the exact request.
Small things that each cost twenty minutes:
Vapi's free numbers. The area code I asked for didn't exist; the error suggested 424, 657, 567, and the code now reads those suggestions and retries. My Gemini key. It came back 403 PERMISSION_DENIED for a billing reason, so the voice-over is my own voice. A shell script. It split a variable in bash and not in zsh, and passed "dialtone to-human" as a single argument. Nothing was dialled, but only by luck. Two logins. A CALL-E login link and a GitHub device code both expired while I was doing something else. The evaluation humbled me twice.
The first blind run said 82.5%. And 6 machines were treated as humans, the exact direction that manufactures false attestations. The ablation said something I didn't want to hear. Removing the timing features raised accuracy on the simulated set from 0.825 to 0.875. The honest reading is not "timing is useless". In simulated text, timing is written by an author, not measured. Only real calls can say whether timing helps, which is why I kept it and why the live event stream mattered so much. Accomplishments that we're proud of The yesbot call.
The transcript. A machine said "You're speaking to the right person," and then "That's confirmed… You're all set," to another machine. The reports. CALL-E recorded success. DIALTONE, which I wrote a few hours earlier, read the same call and said refused. What it proves. The whole idea of the project, working on a real phone line, against a bot designed to fool it. The rest, measured rather than claimed:
Detection, blind set 1 (first run): 82.5% accuracy, macro-F1 0.83. Typical calls 96%, hard calls 54%. 1 human treated as a machine, 6 machines treated as humans. Detection, blind set 2: 95.8%. It is probably an easier set, so I read 83–96% as a range, not a score. Attestation, blind set 2: 0 false human_attested. All eight disagreements were in the cautious direction. I define an over-trusting error as one where the gate's label is more trusting than the reference, with the ordering $$\tau(\texttt{refused}) = \tau(\texttt{unverified}) < \tau(\texttt{agent_asserted}) < \tau(\texttt{system_asserted}) < \tau(\texttt{human_attested})$$ and there were zero. It only caught 1 of 4 refusals on that set, and the report lists why. Live calls: 4 real CALL-E calls to a voicebot line so far [update after calls 5–6]. Every one had a pickup collision. The behavioural detector identified the machine in every one, once with only 59% confidence, which the gate treated as not enough. CALL-E called one bot a human; DIALTONE didn't attest it. CALL-E marked the yesbot booking done; DIALTONE refused it. Tests and packaging: 17 tests, including one for each bug a live call found. The submission package passes the repository's own validator. And the honest one: no business's real voicebot has been called by this yet. The voicebots were mine, on a free number, playing a restaurant.
What we learned Measure the thing you're afraid of. The number I cared about wasn't accuracy. It was machines treated as humans, because that's the path to a false human attestation. Naming that error, counting it, and giving it its own veto changed the design more than any feature did.
A self-report is a claim, not a fact. CALL-E's model told me a voicebot was a human. My own detector told me the same thing an hour later. The fix wasn't a better model. It was refusing to let any single view of the call grant the most trusting label on its own.
Real telephony has a first second, and it matters. In simulation, turns take turns. On a real line, two agents built to be helpful both speak first. A protocol for agents has to be designed for that collision, not for the transcript.
Blind means blind once. The moment I read the errors of an evaluation set and changed code, that set stopped being a test. So I archived the first run, labelled every later one "not blind", and had a new set written by an author who couldn't see anything. It's more work. It's the only version of the number I'd be willing to defend.
What's next for DIALTONE Listen-first. The live calls showed the handshake can't land if both sides speak at pickup. DIALTONE v2 needs the caller to wait for the greeting, and that belongs in CALL-E's call settings rather than in a prompt.
Live detection. The event stream is already live. DIALTONE can classify during the call today. The next step is acting on it: a mid-call hook that lets the caller switch mode on DIALTONE's decision instead of its own guess.
The inbound side. The same detector already runs from the voicebot's side (dialtone line inbound) and classifies the caller. For a company answering a million calls a day, that's the version that matters: knowing which callers are agents, and answering them in machine mode.
A standard someone could own. DIALTONE v1 costs an inbound voicebot three sentences in its system prompt. The protocol is only useful if both sides speak it, which is why it should belong to the platforms at both ends of the call, not to me.
Log in or sign up for Devpost to join the conversation.