CrisisDesk

Four specialized AI agents triage, allocate, audit, and dispatch emergency resources in real time — then a built-in benchmark proves they beat a single-agent baseline on decision quality.

Comment

Inspiration

Emergency dispatch is one of the hardest real-time resource-allocation problems there is: scarce resources, competing priorities, hard rules that must never be violated, and no time to think twice. A single dispatcher — or a single AI model — has to assess severity, check what's available, enforce priority, and write instructions, all at once, for every incident that lands on the desk.

The question we wanted to test: what if instead of one model doing everything, four specialized agents each owned one piece of the decision — and checked each other's work before anything got dispatched? Track 3 (Agent Society) asks for a measurable efficiency gain over a single-agent baseline, not just a demo, so we built the benchmark first and let the numbers argue for the architecture.

CrisisDesk is explicitly a simulation with invented rules — not a real 911 system — built to make that argument honestly.

What it does

Four agents, one pipeline:

  • Triage scores every incoming incident's severity (0–10) and priority tier
  • Allocator proposes a specific resource (ambulance / fire truck / rescue boat), weighing proximity and type
  • Auditor checks the proposal against a hard rulebook — no double-assignment, no LOW-tier resource grabs while a CRITICAL incident is unresolved, resource type must match — and can reject and force a revision
  • Field Liaison writes the actual dispatch instructions once approved

Incidents that arrive together are triaged concurrently (not one-by-one) and allocated in priority order, so the agents are genuinely forced to decide who deserves the scarce resource more. When two incidents tie on severity and need the same resource type, the system picks whichever has the fastest reachable resource — and logs why, visible right in the live feed.

A built-in benchmark runs the identical scenario through this four-agent pipeline and a single-agent baseline, then scores both 0–100 on rule compliance, resolution rate, and escalation accuracy — a real, computed quality score, not a vibe check.

How we built it

Python + FastAPI backend, SQLite for the full audit trail (incidents, resources, every agent message, conflicts, dispatches, benchmark runs), a vanilla HTML/CSS/JS dashboard with a live WebSocket feed and a canvas-rendered map, all talking to Qwen models through Alibaba Cloud's Dashscope endpoint. Deployed on an Alibaba Cloud ECS instance.

Challenges we ran into

The project went through two hardening passes, and both surfaced real bugs.

Early build:

  • A global uniqueness constraint caused silent resource collisions. Resource IDs were unique across the whole database rather than scoped per run, so every run after the first silently failed to seed its resource pool — INSERT OR IGNORE swallowed the error and the agents saw an empty pool without any indication why. Fixed by scoping the constraint to (resource_id, run_id) and prefixing every ID with the full run ID.
  • The map showed incidents that weren't real. The frontend pre-seeded placeholder incident markers before a run started, then real incidents arriving over the WebSocket got drawn on top of the placeholders — producing duplicate dots with the wrong color and wrong status. Fixed by removing all client-side seeding; the map now only ever reflects incident events the backend actually emits.
  • The benchmark was measuring the wrong thing. Early versions scored multi-agent vs. single-agent on wall-clock response time. Multi-agent always lost, because it makes four sequential Qwen calls per incident against one for the single agent — that's measuring network latency, not intelligence. Replaced with a Quality Score (0–100) based on decision correctness, keeping response time only as a clearly-labeled, honestly-caveated secondary metric.
  • Escalated incidents were being marked resolved. resolve_incident() was getting called even when no resource was actually found. Split into a separate escalate_incident() path so the two outcomes can't be conflated on the map or in the stats.

Hardening the benchmark further:

  • The comparison still wasn't fair. Even after the fixes above, incidents were processed strictly one at a time, so the single-agent baseline behaved identically to the multi-agent pipeline — neither was ever actually forced to prioritize. Rebuilt the pipeline so incidents arrive as a true simultaneous batch: multi-agent triages everything concurrently and allocates by priority, while the single-agent baseline deliberately just works through them in arrival order. That asymmetry is what finally made the benchmark mean something.
  • A resource-type fallback bug caused nonsense dispatches. When the required resource type wasn't available, the code silently fell back to offering any type as a candidate — resulting in an ambulance being dispatched to a structure fire. Removed the fallback and added an explicit hard rule requiring type matches.
  • The quality scorer had a dead formula. One scoring component evaluated to 1.0 no matter what happened, quietly inflating scores. Also found the priority-violation detector was flagging false positives whenever any critical incident was open, regardless of whether it actually needed the same resource type as what got dispatched. Both fixed to check genuine same-type contention only.
  • The Auditor occasionally hallucinated a rejection, contradicting the ground-truth "already assigned" flag it was explicitly given. Strengthened the prompt to make that flag unambiguous.
  • Triage severity was inconsistent — the same incident type could score CRITICAL in one run and URGENT in another, because the model had no anchor point. Fixed by passing each incident type's baseline severity into the prompt as a reference.
  • The Stop button didn't actually stop a benchmark comparison — traced to the run/compare endpoints never being wrapped in a cancellable task in the first place. Fixed by tracking them as real asyncio.Tasks the Stop endpoint can reach.
  • "Clear" didn't actually clear. Resetting a live session only detached it in memory — the data was still in SQLite, so it reappeared after a page refresh. Fixed with real deletion across every affected table.

Accomplishments we're proud of

A benchmark that actually, honestly demonstrates the Track 3 requirement: the multi-agent pipeline measurably outperforms the single-agent baseline on decision quality, with a documented, logged reason for every escalation and every tie-break — not just a claim in a slide.

What we learned

Multi-agent value doesn't come from having more models — it comes from structural asymmetry: giving one role (the Auditor) the specific job of catching what the others miss, and forcing incidents to genuinely compete for scarce resources instead of processing them in a queue where competition never actually happens. We also learned how easy it is for a benchmark to quietly measure the wrong thing (latency instead of quality, or a scoring formula that's secretly a tautology) unless you keep re-checking what your numbers are actually proving.

What's next

Real geospatial data and traffic modeling, more agent roles (e.g. a resource-replenishment planner), and multi-city coordination where separate dispatch centers negotiate cross-jurisdiction resource sharing.

Built With

Share this project:

Updates