Locked Door

Inspiration

Every heat emergency produces the same story. A county publishes a list of cooling centers. A facility changes its hours, moves its entrance, stops taking pets, or was never activated this season. Nobody updates the PDF. Someone in 105-degree heat follows the listing and finds a locked door.

The infuriating part is that the current fact almost always exists. It is sitting behind a phone number, known to whoever answers, and completely invisible to every search API and every scraper. A phone call is the only sensor that can read it.

But "an AI that calls places" is not a product. The product is the policy: which facts are worth a call, what counts as evidence, when to say you do not know, and what you are allowed to publish on your own authority.

What it does

It takes a published emergency-resource directory and turns a bounded number of phone calls into a reviewable changeset.

1. It decides what to verify. Every (facility, field) pair is scored on harm if stale x probability of being stale x whether a phone call could even observe it. Staleness probability comes from field-type volatility priors and age since last verification. A greedy selection with diminishing returns then picks facilities, not fields, because one call verifies several things at once.

2. It calls, within a budget. Realistic call outcomes: connected, IVR menu, voicemail, no answer, disconnected line. Retries with backoff and a per-facility attempt cap.

3. It grounds everything. Every extracted field — open now, hours, pet policy, accessibility, intake requirements, capacity status — must point at a span of the transcript. If no span supports it, the value is unknown. That is enforced in code, not by convention, and the citation is visible in the UI.

4. It publishes a delta, not an overwrite. Contradicted or low-confidence fields go to a review queue. What survives becomes a time-stamped changeset with old value, new value, evidence, confidence and verified-at, awaiting human approval, with full per-field history and freshness decay.

How we built it

Vanilla JavaScript, static files, no CDN and no backend. The risk model, planner, dialogue simulator, extractor, publisher and evaluator are separate modules with one shared codepath — the browser and the headless test harness produce identical numbers.

No real calls are placed

The call transport is an interface with two implementations: a real CALL-E telephony adapter, wired but disabled, with its request shape documented; and a deterministic simulated facility with ground-truth state that plays realistic conversations. The UI says calls are simulated at all times and 0 external network requests occur across a whole session.

We found and fixed a real defect here: placeCall() on the live adapter was async, so it returned a rejected promise rather than throwing. An unawaited call would have sailed straight past the refusal and only surfaced if someone was listening for it. On the one boundary that must never be crossed quietly, that is not good enough. It now throws synchronously.

Accomplishments we're proud of

Measured over a 25-call batch against the simulator's ground truth:

• precision 94.4% (67/71), recall 44.7% (67/150), unknown rate 52.7%
• false-publish rate 0.00% — the extractor produced 4 wrong values and the router caught all 4; zero were auto-published
• risk-ranked 49.31 harm units vs oldest-first baseline 39.35, a +25.3% lift — realized against ground truth, not planned. The lift holds across budgets (+42.1% at budget 10, +32.4% at 15, +25.3% at 25) and correctly decays toward zero as the budget approaches the full directory, which is what a real optimizer should do.
• citation audit: 71/71 pass, 0 failures, using an independent auditor that re-slices each span out of its retained transcript rather than reusing the engine's own gate

Honest limits

Recall is 44.7%, and that is the real story of this product. Over half of attempted facts stay unknown, driven by unreachable lines, staff who genuinely do not know, and ambiguous speech ("depends on staffing, it varies"). That is the designed trade, but nobody should read 44.7% as a good number in isolation.

The 0.00% false-publish rate is the rate at which the system publishes a wrong value on its own authority. If a human then blanket-approves all 34 review rows unread, it becomes 5.63%. We show both on screen, because approveAll() simulates exactly the rubber-stamping the review queue exists to prevent.

capacity_status recall is 24% — it is the most volatile field and the least reliably answered, so it burns budget for little return. open_now precision is 84%, our weakest field, because stale voicemail greetings ("closed for the season") get cited as evidence about today; they are correctly low-confidence and routed to review, but the greeting-as-evidence rule is the softest part of the extractor.

The extractor is rule-based, not a model: deterministic and auditable, but it misses phrasings outside its rules — which shows up as unknown, never as a wrong answer. And the directory is synthetic; the schema is realistic, the facilities are not.

What we learned

Once "no citation means unknown" was enforced in code rather than encouraged in a prompt, most of the product's other decisions fell out of it. The review queue exists because the rule produces contradictions. Freshness decay exists because a citation has a date. The risk model exists because unknowns are expensive and calls are finite.

What's next

Wire the real CALL-E adapter behind an explicit operator opt-in with recorded consent, replace the rule-based extractor with a model that still cannot emit an uncited value, and pilot against one real county directory with a human verifying every publish.

Built With

• canvas
• civic-tech
• evidence-grounding
• information-retrieval
• javascript
• optimization
• public-safety
• simulation
• voice-ai