disasterlens-agent

DisasterLens: an AI agent that finds your missing family across disaster shelters in any language, with any spelling of the name, and a human verifier on every match.

Comment

Inspiration

When Hurricane Elena scatters Houston's evacuees across forty shelters, María — 68, Spanish-speaking — can't find her grandson Carlos. The Red Cross has a registry, but Carlos is logged as Carlitos M. at one shelter, Carlos Martinez at another, and C. Martínez at a third. María doesn't speak English. She just needs to know where Carlos is.

Family reunification at scale is real work. After Hurricane Katrina, NCMEC fielded 34,045 calls and reunited 5,192 missing children. The American Red Cross, ICRC Restoring Family Links, NCMEC's Unaccompanied Minors Registry, NamUs, and UNHCR's BIMS all operate today on a primarily-English, primarily-manual matching workflow that systematically under-serves the Spanish-, Arabic-, Vietnamese-, and Chinese-speaking communities most likely to need it after a US disaster.

DisasterLens is the multilingual matching engine those workflows don't have natively. Crucially, it is designed to slot in alongside the existing programs, not replace them — it hands off to NCMEC UMR for unaccompanied minors, to ICRC RFL for cross-border cases, to NamUs for unidentified remains, and to UNHCR BIMS where biometric ID is the right primitive.

What it does

An AI agent — built on the Google Agent Development Kit with Gemini 2.5 Flash on Vertex AI — helps families reunite across languages, name spellings, and shelter rosters. A human verifier approves every match before anything reaches a phone.

  • A seeker writes (or calls) in any of six languages — English, Spanish, Arabic, Vietnamese, Chinese, French. The chat UI auto-detects, handles RTL natively, and the agent replies in the seeker's own language.
  • The agent searches four Elasticsearch indices — shelter rosters, missing-person reports, open reunification cases, and multilingual social posts — through Elastic's Agent Builder MCP server over Streamable HTTP, using five compounded matching strategies:
    1. Standard analyzer with a nickname synonym_graph (Carlos ↔ Carlitos, Mohammed ↔ Muhammad)
    2. Double-metaphone phonetic
    3. ICU transliteration + diacritic folding
    4. Deterministic variant expansion (Arabic romanization, Vietnamese fold, name-order swap)
    5. Multilingual semantic kNN via E5-small embeddings
  • The agent surfaces ranked candidates to a verifier in a custom React + MapLibre UI, with two policy gates wired end-to-end: disclosure-consent (the candidate must have agreed to be findable) and minor-protection (under-18 matches require explicit guardian-verification, following the 2013 FEMA/NCMEC/HHS/ARC Post-Disaster Reunification of Children doctrine).
  • On approval, the agent drafts a multilingual notification (Spanish usted-form, Arabic with a culturally appropriate greeting) and dispatches via Twilio SMS — with the same gates re-checked server-side as a runtime backstop.
  • Three modalities, one Coordinator agent. A multilingual chat UI, a Twilio voice gateway with DTMF language pick and Polly TTS, and a programmatic API all run against the same LlmAgent. Dial +1 (586) 210-4811 to try the voice path.
  • Standing queries stay active for unresolved cases; a Cloud Run Job watcher re-fires the search as new shelter roster docs arrive — turning the "third minute of a real reunification" (when matches start surfacing) into a live UI event.
  • Resolved cases can be federated to other reunification registries via a PFIF 1.4 XML export adapter, with location coarsened to city level for minors.

How we built it

Three required-tech surfaces:

  • Gemini 2.5 Flash on Vertex AI is the agent's reasoning model AND the Vision second-opinion for photo-vs-photo similarity. Vertex routing forced via GOOGLE_GENAI_USE_VERTEXAI=true. Imported at agent/config.py:11, agent/main.py:23, agent/tools/photo_match.py:131-132.
  • Google Cloud Agent Builder (ADK) powers the entire runtime: LlmAgent for the Coordinator + Intake + Notifier sub-agents, AgentTool wraps the sub-agents as callable Coordinator tools, Runner drives every request, and the long-running-tool pattern (await_verifier) is the canonical HITL gate. root_agent is discoverable for adk dev and adk deploy.
  • Elastic Agent Builder MCP is reached over Streamable HTTP at ${KIBANA_ENDPOINT}/api/agent_builder/mcp. The agent discovers ~21 platform tools at runtime and prefers four custom branded skills — match_person_across_rosters, search_social_mentions, create_reunification_case, register_standing_query — implemented as named Python FunctionTools that internally execute the right Elastic query shape and surface in the trace by name.

Supporting Google Cloud surface: Firestore for the HITL pending-decisions state, Secret Manager for API keys, Cloud Run for the verifier+seeker UI service and the voice gateway (--min-instances=1 for stable judging-week latency), Cloud Run Jobs for the incident stream and the standing-query watcher.

The eval harness lives at evals/score.py. For each held-out case it computes a fused confidence that combines BM25 retrieval, query-token overlap, and age tolerance:

$$ \text{conf}(c) = 0.7 \cdot \left( \min\left(\frac{s_{1}}{12}, 1\right) \cdot \frac{|T_{q} \cap T_{c}|}{|T_{c}|} \right) + 0.3 \cdot \mathbb{1}!\left[|a_{q} - a_{c}| \le 3\right] $$

where $s_1$ is the top-1 BM25 score, $T_q$ and $T_c$ are the query and candidate token sets after variant expansion, and the age term contributes a full +0.3 when ages agree within 3 years (0.12 otherwise). The scoreboard additionally reports a 10-bucket reliability diagram, Brier score, Expected Calibration Error, and a bias-by-script recall gap.

Eval results (50-case held-out, dirty-rosters baseline with 15% realistic registrar errors):

Metric Value
Fused precision @ conf ≥ 0.75 0.87
Hero-subset recall (transliteration / nickname) 0.74
Recall gap across name scripts (Latin / Arabic / Vietnamese) < 0.12
Marginal cost per case (measured Vertex tokens) ≈ $0.005

Challenges we ran into

  • Elastic Cloud Serverless 9.5's inference ingest processor silently drops embeddings. The data generator now embeds client-side and writes the vector into each doc before bulk-loading — same model, same vectors, but a debuggable failure mode.
  • The .es. vs .kb. endpoint split. Agent Builder MCP lives on the Kibana host, not the Elasticsearch one. Cost an afternoon on day 1; agent/config.py now derives .kb. from .es. by string-replace so the gotcha is one less.
  • ADK sub_agents=[] is a one-way transfer. Sub-agents wrapped via AgentTool are callable and return control to the Coordinator; sub_agents=[] is a hand-off the Coordinator can't resume. This distinction is load-bearing for multi-step reasoning.
  • Twilio's 15-second webhook timeout vs the agent's ~30-second runtime. Solved with a TwiML <Redirect> polling chain — each /voice/poll returns in milliseconds, the agent runs in a background asyncio.Task, the caller hears hold music until the result is ready.
  • A2P 10DLC blocks outbound SMS on US trial numbers. Demos go phone-in / spoken-reply only; the Notifier's mock-banner fallback covers the SMS beat visually in the recorded demo.
  • Cloud Run deploy gotchas, all encountered for real on the way in:
    • gcloud builds submit doesn't accept both --tag and --config (mutually exclusive).
    • --config=- doesn't accept stdin; needs a real file path. Now generated with mktemp.
    • uv sync builds the local project as part of resolution, which means Hatchling needs README.md and the package source to exist at that layer. Split into a two-stage sync.
    • Cloud Run's default Compute Engine service account doesn't get roles/secretmanager.secretAccessor automatically — explicit IAM grant required.
  • f-string braces in the Coordinator system prompt. Adding a docstring containing {comparable, ...} made Python try to evaluate it as an expression at import time. Fixed by doubling the braces, then by a smoke test that catches this class of bug.

What we learned

  • Elastic Agent Builder MCP is dramatically more capable than the legacy @elastic/mcp-server-elasticsearch. The ~21 platform tools plus the ability to register custom skills make the difference between "the agent calls one ES query" and "the agent shows 5–8 tool calls per reasoning chain that judges can scroll through."
  • The ADK AgentTool vs sub_agents distinction matters. Wrapping sub-agents as AgentTool keeps the Coordinator in control of the loop; sub_agents=[...] is a one-way transfer. We learned this the hard way when the Coordinator stopped reasoning mid-chain.
  • Verifier-gate-as-product, not as-feature. Once we treated the HITL gate as the showcase capability — consent + minor badges, server-side backstops, triage view, FEMA/NCMEC doctrine cited in the UI — the rest of the UX fell into place. Trust matters more than algorithmic precision in this domain.
  • Multilingual matching is not "translate, then search." The compound analyzer stack (standard + phonetic + translit + nickname graph + semantic kNN) does work no single-strategy approach can do, and Elastic's _explain output makes it visible enough to demo — evals/explain_match.py renders the per-analyzer BM25 contribution as an ASCII bar chart, which is what we use in the recorded walkthrough.
  • Reporting dirty-rosters numbers is more credible than precision on clean fixtures. Real shelter intake produces typos, swapped name order, dropped fields. Reporting "0.93 on clean + 0.87 on dirty" anchors the claim against the obvious "your synthetic data was too clean" objection a thoughtful judge will form.

What's next

  • Practitioner validation. docs/outreach_kit.md is the cold-outreach kit for a 30-minute conversation with a Red Cross volunteer coordinator, FEMA Family Assistance Center alum, or NCMEC communications team member. One quotable line from someone in the field turns this from an AI demo into a field-aware system.
  • Federation handshake. PFIF export exists; the next step is a PFIF import adapter so DisasterLens can ingest from NCMEC UMR / ICRC RFL feeds rather than only emit to them.
  • Biometric tier boundary. For mass-casualty incidents the right primitive is dental / DNA / fingerprint, which is where UNHCR BIMS and NamUs live. DisasterLens should declare the hand-off and integrate the relevant export.
  • Real shelter pilot. The Texas Voluntary Organizations Active in Disaster (TX VOAD) network is the natural first deployment partner; their next tabletop exercise is in autumn 2026.

Built With

Share this project:

Updates