Aftershock

A disaster-struck town run by a society of Qwen agents that split tasks, negotiate scarce rescue resources, and measurably beat a single big model on lives saved per dollar.

Comment 2

Project Story

Inspiration

Months of experimenting with long-running multi-agent simulations taught me an uncomfortable lesson: many "agent teams" are theater. Agents chat, agree with each other, duplicate work, and a single strong model with a good prompt quietly outperforms them.

The interesting question is not can agents collaborate. It is when does coordination let a society of small models match a much larger model, what does it cost, and can you prove it?

Aftershock answers that question in an unforgiving simulated setting: disaster response, where coordination failures have measurable consequences. It asks whether a structured society of Qwen agents can split work, negotiate over scarce resources, and reach the same outcome as a flagship solo model at lower cost — while making every claim inspectable rather than asking judges to take it on faith.

What it does

Aftershock is a deterministic disaster-response simulator and live observatory for Qwen agent societies.

A disaster hits a simulated city. Missions appear on the map: flooded neighborhoods, a collapsed school with people trapped, a hospital running on generator fuel. A society of Qwen agents with six distinct roles must respond: incident commander, medical, fire, rescue, infrastructure, and public communications.

The project has four core pieces:

  • Task division: the commander decomposes each mission into typed work and assigns it by role. Agents can accept, reject, sub-delegate, or escalate with structured reasons.
  • Negotiation under scarcity: ambulances, rescue crews, fire engines, fuel, and generators are limited. Agents compete for them through a typed proposal protocol: resource requests, task handoffs, escalations, and information shares. The engine resolves conflicts atomically every tick, so coordination is a mechanic rather than a chat transcript.
  • Measurement you can audit: the benchmark runs identical seeded disasters four ways — scripted bots, one qwen3-max solo agent, a flat swarm of qwen3.5-flash agents without the protocol, and the structured society (qwen3.5-flash workers with a qwen3.5-plus commander). It scores lives saved, response latency, missions failed, wall time, and cost — and shows a bootstrap confidence interval, a paired sign-test, and statistical power beside every delta, flagging weak effects as suggestive rather than proven. A reproducibility badge confirms the scripted engine replays byte-for-byte.
  • Auditability: nothing is asserted that you cannot inspect. A Decision Receipt chains any contested call — the kernel's ruling and its reason, the agent's own rationale (clearly labelled "agent-stated," never dressed up as ground truth), the token cost, and the recorded outcome. A one-page Evidence Pack ties every headline number to a source file in the repo.

The headline result is about cost-efficiency, not a claim that more agents save more lives. In the four-arm Qwen benchmark, five qwen3.5-flash role workers under a qwen3.5-plus commander save 108.4 simulated lives at about $0.035 per run (~3,070 lives/$). That matches expert scripted heuristics (106.8) and the qwen3-max solo arm (95.6), while delivering over 50% better lives-per-dollar than the solo Qwen flagship.

We then ran the load-bearing test: the identical solo arm on 12 models from 10 families, over 10 paired seeds each. No solo model beat the coordinated Qwen society on lives. Eight frontier models only tied its outcome, and most cost 3–14× more per run. The honest exception is DeepSeek V4 Flash, which tied on lives and beat the society on cost. The conclusion is therefore precise: coordination can substitute for model scale on this decomposable task, and usually does so much more cheaply; it does not magically raise the outcome ceiling.

Two further results keep that conclusion honest. Written doctrine raises team conformance by a credible +0.125 (positive on 6/6 paired seeds, sign-test p=0.031) at no detectable cost in lives. Against the flat protocol-free swarm, however, the society's +8.9-life edge at n=15 is suggestive, not significant: its bootstrap 95% CI [+2.3,+15.4] excludes zero, but the paired sign-test p=0.118 does not. An earlier n=5 estimate of about +28 lives collapsed when we added ten seeds; the benchmark caught its own ghost, and we corrected the headline rather than hiding it. The full evidence trail is in the Evidence Pack.

Recorded runs are replayable, and the simulator has a strict determinism boundary: the same world seed plus the same recorded decisions produces the same outcome byte-for-byte. Qwen inference itself is not claimed to be deterministic. The web observatory lets judges scrub through those records, inspect agent decisions, open a Decision Receipt for any contested ruling, compare arms with confidence intervals and significance tests, branch counterfactual what-if runs, watch live runs, and explore an honestly labelled real-data scenario based on NYC Hurricane Ida.

How we built it

Aftershock is built around a small reusable agent-society kernel in Python.

The kernel provides a deterministic tick loop, typed decision validation, a negotiation protocol, atomic per-tick resolution, role definitions in YAML, a token/cost ledger, and replayable NDJSON run records. The disaster town is the flagship society built on top of that kernel.

The Qwen layer uses cost-tiered cognition:

  • qwen3.5-flash handles routine role decisions.
  • qwen3.5-plus acts as the structured commander/arbitrator.
  • qwen3-max writes after-action reports and higher-level analysis.

Agents output strict JSON through Qwen Cloud compatible chat completions. The simulator validates every decision before it touches the world, and rejected decisions are fed back to agents so they can recover instead of silently corrupting the run.

The observatory is a React and TypeScript frontend served by a FastAPI backend. It streams live runs over WebSockets, replays recorded runs, compares benchmark arms with server-computed confidence intervals and sign-tests, renders per-decision evidence receipts, surfaces scenario provenance, and exposes a reproducibility check that re-runs the scripted engine and confirms identical world digests. The same backend exposes an MCP spectator server so an MCP client can inspect runs or inject live events.

The public deployment runs on Alibaba Cloud ECS using Docker Compose. Caddy terminates HTTPS at aftershock.redoubtlabs.dev and proxies to the FastAPI app.

Real-data scenario: NYC Hurricane Ida

The benchmark is synthetic so every arm faces byte-identical worlds. But Aftershock also ships offline-compiled real-data scenario packs for demonstration, replayable directly in the live observatory.

The headline pack is nyc-ida-2021, built from FDNY EMS Incident Dispatch Data and Fire Incident Dispatch Data via NYC Open Data for the night of Hurricane Ida, 2021-09-01 18:00 to 2021-09-02 06:00 EDT.

The pack uses real incident demand and real first-on-scene latency as the baseline:

  • 2,003 EMS rows in the Ida window.
  • 16.5% held rate.
  • 948 seconds mean first-on-scene.
  • Calm comparison window two weeks earlier: 524 seconds mean first-on-scene and 4.2% held.

The honesty contract is explicit everywhere in the UI: demand and latency are real; mission kinds and severity are mapped; lives at risk are inferred; blockages and outcomes are simulated. The project does not claim to reproduce real casualty outcomes.

Challenges we ran into

The hardest part was making the system measurable instead of merely impressive.

Small models will invent entity IDs, repeat invalid actions, and miss one-tick feedback. The fix was not more prompting alone. We needed engine-side validation, named rejection reasons, and short-term rejection memory in the next observations.

The second challenge was proving the society mattered. If every arm sees a different world, the benchmark is just a story. Aftershock therefore uses paired world seeds: the solo agent, swarm, scripted baseline, and society face byte-identical disasters. Because Qwen output is stochastic, we pair the worlds rather than pretending the LLM calls are reproducible, and report confidence intervals, exact paired sign-tests, and statistical power beside each delta.

That discipline changed the pitch. At five seeds, society appeared to beat the flat swarm by about 28 lives. At fifteen seeds the mean fell to +8.9 and failed the paired sign-test, so we downgraded it to suggestive. Cross-family testing then showed that genuine frontier solo models reach the same outcome ceiling. The durable win is cost-efficiency, not a universal lives advantage for societies.

The third challenge was honesty around real data. NYC Open Data gives real incident timing and response latency, but not real "lives saved" for our simulated missions. The UI had to show that boundary clearly instead of hiding it in a footnote.

Accomplishments that we're proud of

  • A full multi-agent society that is not just a chat room: it has typed roles, typed proposals, validation, auctions, rejection feedback, and measurable outcomes.
  • A benchmark that compares architecture choices under identical seeded worlds.
  • A 12-model, 10-family cross-family panel showing the cost-efficiency win isn't a Qwen artifact: no solo model (GPT-5, Gemini 3.1 Pro, Claude Opus 4.8, DeepSeek, Kimi, GLM, …) beats the cheap coordinated Qwen society on lives. Most frontier models only tie it at 3–14× the cost; DeepSeek V4 Flash is the documented exception that ties the outcome and costs less.
  • An honesty layer judges can audit: per-decision evidence receipts, confidence intervals and significance tests beside every result (weak effects labelled suggestive), a determinism-verified reproducibility badge, and a citable one-page evidence pack.
  • A public observatory where judges can inspect the actual run records instead of trusting a summary.
  • A real-data NYC Hurricane Ida scenario with visible provenance and caveats.
  • A deployed Alibaba Cloud demo using Qwen Cloud models end to end.
  • Negative results published alongside positive ones, including the finding that scripted heuristics with the same protocol remain highly competitive.

What we learned

The strongest lesson is that coordination can substitute for model scale when the work is decomposable and the models are above a capability floor. A society of small Qwen models reaches the same outcome ceiling as frontier solo models, usually at a fraction of their cost. More agents alone are not the answer: the flat swarm remains at the uncoordinated floor, while the structured society gives roles a typed mechanism for exposing and resolving resource contention.

The qualification matters. The measured society-versus-swarm lives edge is suggestive rather than significant, and a very small local Qwen model cannot carry the protocol at all. Coordination is an engineering substitute for some model scale, not a free replacement for capability. The most statistically credible behavioral lever is written doctrine: +0.125 team alignment at n=6, positive on all six seeds, p=0.031, with no detectable lives penalty. The verified numbers and qualification rules are collected in the Evidence Pack.

The uncomfortable lesson is that the protocol carries more of the result than the LLMs do. Well-tuned scripted agents using the same coordination protocol remain competitive. That is not a failure of the project; it is the point. Agent societies need institutions, contracts, and measurement, not just more agents talking.

What's next for Aftershock

Next we want to turn Aftershock into a general benchmark harness for agent societies:

  • More real-data scenario packs from different hazards and cities.
  • A public leaderboard comparing society architectures against solo, swarm, and cross-family frontier baselines.
  • More seeds per result so today's suggestive edges become statistically settled.
  • Better memory loops where lessons are expressed in the agents' actual action space.
  • More MCP tools for external spectators and human-in-the-loop incident injection.
  • A schema-driven society-pack interface so other domains can replace the disaster roles, resources, actions, scoring, and UI vocabulary without replacing the deterministic kernel.
  • Community-contributed role packs and doctrine files.

The larger goal is to make multi-agent systems falsifiable: not "look how many agents are talking," but "here is the coordination mechanism, here is the baseline, here is the measured gain, and here is the receipt for every number."

Additional Info

Qwen Cloud API usage

Aftershock uses Qwen Cloud compatible chat completions for the LLM arms:

  • qwen3.5-flash for worker role agents.
  • qwen3.5-plus for the commander/arbitrator.
  • qwen3-max for the solo baseline and after-action analysis.

The benchmark reports token usage and cost per run, so Qwen usage is visible in the results rather than hidden behind the demo. In the refreshed four-arm benchmark, a full six-agent society run — five flash workers plus a plus commander — costs about $0.035.

Native function calling — implemented and measured. Beyond strict-JSON contracts, the society also speaks Qwen Cloud native function calling: per-role tools, tool_choice="auto", parallel_tool_calls, and a no_op idle tool (aftershock run --arm society --society-tools). We didn't just bolt it on — we benchmarked it on the same paired seeds and found native tools held lives saved within noise (98.2 vs 103.2) but cost ~2× more and ran ~2.5× slower, because the ~1k-token tool schema is re-sent on every one of ~240 agent calls per run. So the cost-optimal default is JSON contracts, with function calling available and measured as an ablation (bench/results/2026-06-13-tool-ablation/). The point isn't to chase the fanciest API — it's to measure when it actually pays.

Alibaba Cloud deployment

The live demo runs on Alibaba Cloud ECS:

GitHub repository

https://github.com/bluntmachetti/aftershock

Built With

Share this project:

Updates

posted an update —

Progress since submission — we turned the measurement harness on the agents themselves.

Since submitting, we ran a disciplined tuning pass on the Qwen agent society, with one rule: before optimizing any metric, prove the metric actually moves a real outcome (lives, missions, or dollars). That bar did most of the work — it killed or shrank most of our backlog before we spent compute on it. A scoping pass skipped four planned levers outright as "outcome-neutral" (they'd move a number that changes nothing), and we caught ourselves about to build a deterministic "guard" whose only effect would have been to inflate a conformance score by hiding behavior the engine already handles for free. We didn't build it. The honest negatives are in the build log; they're the reason to trust the positives.

What survived the bar landed on both axes the society can actually win on. On coordination quality, a paired ablation confirmed the written doctrine raises team conformance on every one of five seeds (+0.156) at no cost in lives — resolving an earlier single-seed scare that suggested discipline might cost lives (it didn't; that was noise). On cost-efficiency — the society's headline pitch (small coordinated models matching a big one for less) — we profiled where the money goes (≈60% of run cost is the same system prompt re-sent every tick), confirmed the provider doesn't cache it, and trimmed the re-sent prompt. The result, measured in a paired A/B: −14% cost per run for a +21% gain in lives-per-dollar, with lives unchanged.

We also found a clean, reusable result on model selection: the one agent role that wouldn't follow its rules from prompting alone turned out to be a model-capability floor, not a prompt bug — a stronger Qwen model fixes it, but at +33% cost for zero lives gained, so we shipped it as an opt-in "high-conformance" operating mode rather than the default, keeping the cost-optimal path intact.

All of it is merged to main (full test suite green, determinism check passing, the frozen negotiation protocol untouched), deployed through our staging→production gate, and live now at https://aftershock.redoubtlabs.dev. The full write-up — including the "a fix that only fools the scoreboard" lesson — is the latest entry in the build log at https://bluntmachetti.github.io/aftershock/.

Log in or sign up for Devpost to join the conversation.