Inspiration
A call task comes back with task_completed: true, a confident score and a tidy structured result. Nothing in that payload tells you whether the callee actually said the thing the result claims, whether the agent told them it was an AI, whether "please don't call again" was honoured, or whether the conversation was a wall of six-second silences. Every team that ships CALL-E into a real workflow ends up writing this review step by hand. The roadmap lists it (apps/web/call-review-console) and the lane was empty.
What it does
A read-only review desk over terminal CALL-E call snapshots. It never places a call; no call-creation code path exists.
- Evidence. Every leaf of
structured_resultis matched against the transcript. Numbers must be spoken; "9 a.m." folds to "9am"; enum and boolean fields are marked needs reading rather than keyword-matched, because "did they agree" is not a string search. An optional Gemini pass cites the supporting turn per field. - Schema. The result is validated against
metadata.result_schemawhen one is present. - Timing. Response latency p50/p95, silences over 4 s and overlaps, all derived from
transcript_turns[].offset_seconds. No audio needed. - Compliance. AI disclosure and whether it was the first turn, stop and opt-out requests and whether the agent actually stopped, card or ID numbers read aloud.
- Verdict. approve / needs_human / reject with one-line reasons, a recorded human disposition, and a benchmark across every call on file.
- Ingest. A webhook receiver for terminal events, with an optional shared token since deliveries are unsigned, plus an opt-in fetch by id.
How I built it
Python 3.12, FastAPI, jsonschema, one static HTML page. The official calle-ai 0.7.0 SDK and REST shapes (recipients[].attempts[].transcript_turns[], completion_confidence) drive the fixtures. Five fictional calls on reserved +1 555 01xx numbers each exercise one rule. Deployed on Cloud Run from a ten-line Dockerfile. A paired skills/call-review skill documents the same checks for an agent that reviews calls without the console.
Every check is deterministic and explainable. The model pass is optional, and when it runs it cites turns rather than asserting.
Challenges I ran into
The API does not let a reviewer list their own calls. GET /v1/calls returns 405, so the console ingests via webhook or fetch-by-id instead. The API reference pages 404, so the payload shape came from SDK docstrings rather than documentation.
Offsets are turn start times only. Latency is therefore agent-start minus previous-callee-start. The console says exactly that on screen instead of pretending to have end times it does not have.
The first evidence matcher rejected a perfectly good call because "9 a.m." did not match "9am". That is why time folding exists, and why yes/no fields are never keyword-matched.
Four rounds of adversarial security review, and I was wrong in a different way each time. The reviewer on the PR found path traversal in the call-id parameter and stored XSS through an onclick attribute, where my escaper covered & < > but not quotes, so x',alert(1),' executed. Round two: the webhook accepted payloads when the shared token was unset, redaction only happened at render time rather than before persistence, and my newly mandatory auth broke the documented fixture demo with a 503 — fixed with a Jupyter-style per-process ephemeral token. Round three: my phone redactor matched \+\d{7,15}, which is E.164 and nothing else, so a transcript saying (555) 010-0123 still wrote the number to disk; rewriting it loosely then broke on timestamps, because 2026-09-04T15:03:06Z is also digits with separators, and a decimal guard I added to protect 12.5 accidentally shielded dotted phone numbers too. Round four found that my own regression fixtures used email addresses on registrable domains, which is precisely the class of data the sanitizer exists to keep out of the repository.
None of that was visible from the demo. All of it came from someone reading the diff adversarially, and the tests that now exist are the shape of those failures.
Accomplishments that I'm proud of
A review step that works on any call ever made, with no contract to write and no stack to run. It fails closed on unknowns rather than approving what it cannot check. And the security work is in the public history: every blocker raised was closed with a test that reproduces it, not with an argument.
What I learned
Confidence scores are the agent grading itself. Only a supporting turn is evidence, which is why the console will mark a field needs reading rather than claim a match it cannot justify.
Compliance failures are cheap to detect from text and expensive to discover later. A missing AI disclosure or an ignored stop request is a few lines of deterministic checking, and the cost of finding it after the fact is not a few lines of anything.
And redaction is much harder than it looks. A timestamp and a phone number are both digits with separators, and the difference between them is the entire problem.
What's next for Call Review Console
Per-turn end times when the API exposes them. Calibration of completion_confidence against verified outcomes across a labelled set, so the score becomes worth something. And a calls.list view the moment that endpoint exists.
Log in or sign up for Devpost to join the conversation.