Verity

Inspiration

Businesses are handing scheduling calls to AI voice agents, and those agents return task_completed: true when they hang up. But that's a claim, not proof. If the caller said "3… no wait, 3:30" and the agent booked 3:00, the record still says "done." We started calling these ghost bookings — outcomes that look finished and aren't. Verity is the layer that refuses to let a completion claim become a real calendar write until the transcript, the calendar, and — when needed — the caller all actually agree.

What it does

Verity is a verification control plane that sits between a phone agent's result and any real-world action. For every call it:

1. Re-fetches the call from GET /v1/calls/{id} — it never trusts the webhook body — and reads the transcript independently with a deterministic date grammar. No LLM ever picks the value.
2. Runs a pure decision gate: seven required-evidence checks (E1–E7). CALL-E's own task_completed is E1 — necessary, never sufficient. An ALLOW also needs an independently parsed value that exactly matches what would be written, an explicit caller confirmation that followed the agent restating the full date and time, no open ambiguity, and a fresh re-check that the slot is still ours.
3. On any failure it BLOCKs with a stable reason_code, and where a concrete value exists it sends a second-channel SMS quoting the transcript's value — not the agent's. A YES reply, or an operator, is the only other path to a commit.
4. Learns. A caught failure pattern is captured as a PII-stripped regression fixture that auto-fires on future calls, so a phrasing that fooled the system once gets stricter treatment automatically — with no human in the loop the second time.

The demo runs three recorded scenarios: a clean confirm (ALLOW), a self-correction the agent missed (BLOCK → SMS → recovered, and a fixture is created), and a brand-new wording that the learned fixture catches on its own. A read-only dashboard shows five metrics — including ghost bookings prevented — and the full append-only event trail for every verification.

How we built it

• Core: TypeScript / Node 20 / Fastify — one stateful process with SQLite (WAL) + an in-process single-flight queue + Server-Sent Events. Deliberately not serverless; the gate needs a consistent local view of the calendar.
• The gate (decide()) is a pure function with zero I/O and a total try/catch: same input → same output, and any thrown error, missing field, or dead dependency returns BLOCK. commit_booking is reachable from exactly one place — an explicit ALLOW.
• Determinism everywhere: a seeded calendar sandbox, recorded synthetic transcripts, Luxon for all date math, and a verity demo run that executes the whole judge script with fixed timing.
• Audit: verification_events is append-only, enforced by a SQLite trigger; rows carry turn indexes and field names, never quoted transcript text.
• CALL-E integration: POST /v1/calls, webhooks with a CALL-E-Event-Id-vs-body-id check plus an opaque path nonce, GET /v1/calls/{id} re-fetch, idempotency keys, and a single reconciliation poll if the webhook never arrives.
• We also ported the gate logic to Python with matching test verdicts and submitted it as a skill to CALLE-AI/awesome-phone-call-agents (PR #364, merged).
• Deployed on Railway via a Dockerfile that compiles the native SQLite module from source.

Challenges we ran into

• ghost_booking_prevented shouldn't depend on which rule fired. We moved it to a property of the evidence — parsed value ≠ target while task_completed is true — so it's counted correctly even when a different BLOCK reason wins the ordered rule table.
• The repair almost booked the wrong slot. When the corrected time lands on a different slot than the one being held, the repair channel now re-checks availability, holds a matching free slot, releases the old hold, and re-points the booking.
• better-sqlite3 had no prebuilt binary for the target runtime, and the newest major dropped prebuilds entirely — pinned an older version and shipped a compiler toolchain in the build image so it builds from source on deploy.
• CALL-E webhooks are unsigned. The entire "never act on the webhook body, always re-fetch and gate on that snapshot" design falls out of not being able to trust the delivery.
• Parser edge cases — "9:00 AM" was producing a phantom "00 AM" time candidate; fixed with a negative lookbehind in the time grammar.
• A live test call surfaced a region constraint (English calls aren't supported for some countries) — exactly the kind of real-world failure a verification layer should surface rather than silently swallow.

Accomplishments we're proud of

• The single hard invariant holds by construction: no calendar write can be triggered by an unverified task_completed claim alone, with a test that proves a high-confidence claim + non-null structured result + no confirming turn still blocks.
• The learning loop works end to end: a failure caught on one call auto-blocks a differently-worded call later, with zero manual intervention.
• Fully deterministic — the demo produces an identical event stream every run, no network luck required.

What we learned

The interesting gap in agent systems isn't "did the task run" — it's "did the right thing happen." task_completed: true is an unverified transaction, and treating it like one changes the whole architecture: fail-closed by construction, an independent re-derivation of the value, a second channel for anything ambiguous, and an audit trail you can't edit. Deterministic replay turned out to be the thing that made all of it both testable and demoable at once.

What's next for Verity

• More verticals beyond scheduling — any agent completion claim that triggers a real action.
• Signed webhooks / stronger provenance if CALL-E adds them.
• Confidence-weighted fixture retirement with false-positive tracking.
• An operator queue UI for the needs_human cases.

Built With

• better-sqlite3
• call-e-api
• docker
• fastify
• luxon
• node.js
• pino
• python
• railway
• react
• server-sent-events
• sqlite
• typescript
• vite
• vitest
• zod