EMResponse

A real-time incident operating system that turns live calls, checklists, and scene media into continuously updating triage, dispatcher-approved guidance, and coordinated responses.

Comment

Inspiration

Emergency dispatchers and EMS crews operate in resource-constrained, high-stakes environments where the “right decision” depends on incomplete information arriving under stress. In the U.S., an estimated 240 million calls are made to 9-1-1 each year, yet a national staffing survey covering 2019–2022 found an average ~25% vacancy rate in 911 centers, effectively leaving one in four positions unfilled.

Even when responders do everything right, the system behind them can still be brittle. A 2024 survey highlighted the compounding effects of logistical inefficiencies and technological gaps in emergency communications centers, where fragmented tools and outdated workflows make it harder to maintain a coherent, continuously updated incident picture. Downstream, resource bottlenecks can translate into harm: a peer-reviewed study found that longer ambulance offload times for chest-pain patients were associated with higher 30-day risks of death and ambulance re-attendance, linking operational delay to worse outcomes.

Our goal is to reduce preventable harm in the first minutes by guiding bystanders through evidence-based actions and giving dispatchers an always-current, structured incident view that improves prioritization and response.

What it does

Real-Time Incident Operating System

  • A single live “incident room” replaces fragmented updates across tools.
  • Dispatchers get one command view: ordered timeline events, status controls, pinned guidance, location, and media playback.
  • Built for live coordination: reconnection, replay, and consistent state for everyone in the incident.

Continuous Multi-Modal Triage Intelligence

  • The AI continuously ingests transcript chunks, checklist signals, scene cues, and available context to classify likely emergency categories and update as new evidence arrives.
  • It computes live severity and risk trends (low to critical) and flags “missing critical data” so dispatchers know what to ask next.
  • Outputs include confidence and safety checks; guidance is designed to support dispatcher-assisted care workflows.

Controlled Escalation

  • Suggested instructions can be approved, edited, pinned, unpinned, or rejected by dispatchers, keeping accountability with the human operator.
  • For complex incidents, the system can run literature-backed retrieval with citations and uncertainty notes.
  • This is explicitly built for real-world constraints where staffing shortages and technology gaps compound operational risk in emergency communications centers.

Response Orchestration

  • Consent-aware medical matching: filters candidates by consent/role, ranks likely profiles, and surfaces only what’s appropriate for safer decisions.
  • Nearby helper orchestration: alerts eligible nearby users, collects responses, tags capabilities, and connects approved helpers into the flow.
  • Secure voice/video escalation: server-controlled call state + role-based access so only authorized participants join at the right time.

How we built it

Multi-Client Frontend

  • Two surfaces, one versioned incident event envelope with shared schemas and backward-compatible evolution.
  • Dispatcher web console: Next.js and TypeScript for event-sourced timeline rendering, triage packets, guidance approve/reject/pin workflow, dispatch actions, resource selection, signed-media playback.
  • Bystander Android app: Kotlin and Jetpack Compose for session bootstrap, multimodal capture (camera, mic, GPS), streaming ASR transcript chunks, guided voice interaction, live incident state sync.

Edge Backend

  • Cloudflare Worker API for low-latency edge ingress, incident RPC routes, AI inference routes, call-control endpoints, media signing, helper orchestration, consent-aware medical matching.
  • Durable Object as per-incident single-writer coordinator for total-order event log, websocket fanout, replay cursors, acknowledgement tracking, lifecycle transitions.
  • Cloudflare R2 for media plane with signed PUT/GET, object metadata validation, and content-type/size enforcement.

Safety and Reliability

  • Strict request validation with unknown-key rejection, schema checks, and fail-closed handlers.
  • Role-based access control for dispatcher, bystander, helper with route-level authorization and capability gating.
  • Deterministic call flow state machine with explicit transitions and idempotent handlers.
  • Correlation IDs, audit trails, redaction-aware logs, and traceable incident-level observability.

AI Decision Support

  • OpenAI structured triage summarization producing typed outputs: triage category, urgency, confidence, reason atoms, dispatcher packet, follow-up questions.
  • Safety policy layer splitting auto-display guidance from dispatcher-reviewed guidance with guardrails for non-standard instructions.
  • Real-time incident analytics over streaming events: risk scoring, priority trend, missing-critical-field detection, alert triggers, recommended next actions.

Retrieval and Vision

  • Novelty and out-of-distribution detection gating advanced behaviors and retrieval escalation.
  • Perplexity Sonar RAG pipeline with query shaping, retrieval, citation parsing, evidence normalization, and structured evidence packets.
  • Video frame VLM assessment emitting structured scene-risk signals and hazard flags from sampled frames.
  • All models outputs schema-validated and contract-checked before timeline writes.

Incident Communications

  • WebSocket channel with ordered event delivery, replay via cursors, and ack semantics for reconnect correctness and at-least-once convergence.
  • Zoom Video SDK integration with server-issued join tokens, role constraints, and consent/lifecycle gating.
  • Nearby helper flow as timeline events: alerting, profession tagging, dispatcher accept/revoke, listener invites, helper resource requests.
  • Consent-aware medical matching as timeline events: ingestion, consent and proximity filtering, similarity ranking, deterministic match-mode selection.

Data Pipeline

  • Medical ingest service mapping CSV batches into canonical normalized profiles with strict validation, reject reporting, and schema enforcement.
  • Drill tooling and fixtures for reproducible incident simulations, novelty checks, schema gates, and replay tests.
  • Shared type contracts to keep clients and backend aligned and force breaking changes to fail fast.

Challenges we ran into

Uncommon Emergencies

  • Rare presentations did not match common triage patterns and could be mis-scored as the nearest common case.
  • We added Mahalanobis-distance novelty scoring to detect out-of-distribution incidents and used it to gate Perplexity Sonar RAG, producing citation-normalized evidence packets.

Real-Time Analytics on a Live Incident Stream

  • Analytics had to run on streaming, partial inputs: transcript chunks, checklist events, media metadata, call-state transitions.
  • We treated the incident as an ordered event log in the Durable Object and recomputed risk level, missing-critical-data flags, and alerts from canonical state using sequence numbers, replay, and acknowledgements.

Zoom Video SDK Call Lifecycle

  • Failures occurred under reconnects and state transitions: token expiry, join races, duplicate joins, consent gating, incident teardown.
  • We issued join tokens server-side bound to incident ID and role, enforced lifecycle gates, and made join/leave handlers idempotent so retries do not corrupt session state.

What we learned

  • Event-sourced incident state and schema versioning to keep clients and backend consistent
  • Real-time updates with WebSockets using replay and acknowledgements for reconnect correctness
  • Long-tail medical case handling with novelty detection and retrieval-gated RAG for citation-backed evidence
  • Safety-first AI integration with RBAC, strict validation, and dispatcher review gates for non-standard guidance
  • Multimodal signal extraction from transcript streams and sampled video frames with structured outputs and latency constraints
  • Third-party service integration under lifecycle edge cases, especially Zoom tokening, consent gating, and retry-safe joins

What’s next

Government-Backed Rollout and Adoption

  • Partner with cities, campuses, and public agencies so the bystander and helper network has enough density to matter. More adoption means faster helper routing, better coverage, and stronger incident outcomes.

Low-Connectivity and Offline Operation

  • Add degraded-mode workflows for low data access areas: store-and-forward event buffering, adaptive media sampling, transcript compression, and audio-only fallbacks so incidents still run when bandwidth collapses.

Multi-Hop Evidence Retrieval

-Upgrade the Perplexity Sonar RAG path from single-pass retrieval to multi-hop retrieval across linked claims, contraindications, and comorbidities, producing chained, citation-backed evidence packets instead of isolated snippets.

Stronger Reasoning and Calibration

-Move from one-shot triage to iterative decision support: uncertainty-aware updates, conflict detection across sources, and targeted follow-up questions that reduce overconfident guidance.

Built With

Share this project:

Updates