Breathing Room: crowd pushing and minutes to danger

A crowd-crush early warning that reads crowd density and pressure in people from ordinary CCTV, predicts the next minutes with the physics of crowd flow, and gives a deadline before anyone falls.

Comment

Inspiration

At Itaewon, Seoul, on 29 October 2022, the first of 11 emergency calls about the crowd came at 18:34. Police investigators later estimated, from security footage, about 8 people per m² in the alley at 22:15 and 9 to 11 per m² by 22:25. 159 people died. Cities now count heads on CCTV. A count says how full a street is. It cannot tell a calm, packed crowd from one where people have started to push, and it cannot say how much time is left. Breathing Room measures both.

What it does

For every zone it measures three numbers from video: density in people per m², a pushing index from crowd motion, and the minutes until the zone reaches 4.5 people per m² at the current inflow. When a zone needs attention, an agent (gpt-oss-120b on Amazon Bedrock) checks that the camera has not moved, counts the crowd a second way, finds where in the zone people are pushing, measures the exit flow, and forecasts every action the venue allows. It puts one proposal in front of a steward: a sentence to read over the radio that says where the pushing is, a camera crop of that spot, the predicted peak with and without the action, how far off that forecast was when tested, and how many people the action would hold outside. An action only counts as safe if it keeps both the zone and the waiting area before the gates out of danger, so it cannot clear the zone by moving the crush to the pavement. The steward approves or rejects. Ninety seconds later the system measures again, and escalates if the curve has not bent. After the evening, the safety officer reads a report generated from the hash-chained record, which states first whether the record is intact. Other agents can read the same measurements through a read-only MCP endpoint; approvals stay with the steward.

How a zone that needs attention reaches the steward

From measurement to decision. OpenCV 5 measures every zone; deterministic tiers open an episode; the agent investigates with its tools, and code checks every proposal before a steward approves it. Ninety seconds later the system measures again.

How I built it

  • OpenCV 5 vision: frames reduced once with resize (area) and undistorted with remap at half size, head detection (cvtColor, inRange, connected components), DIS optical flow, head-plane homography (USAC MAGSAC), Helbing's crowd fields with GaussianBlur on a 0.1 m grid, people counted across a gate line, pushing hotspots from connectedComponentsWithStats on the floor grid cut out of the image with warpPerspective, DM-Count on the OpenCV 5 DNN engine as a second counter, and a two-stage camera watchdog: findTransformECC (homography model, still-pixel mask) every 3 s, confirmed by ALIKED + LightGlue from the OpenCV 5 features module.
  • Physics: the continuity equation over each zone (dN/dt = Q_in − Q_out) gives the inflow and its trend from measured quantities; integrated against exit capacity measured on 24 real bottleneck runs, it gives minutes to danger and the effect of each action.
  • Agent: deterministic tiers decide when to investigate; the model picks tools, compares actions and writes the radio message. The safeguards are code, not prompt: a proposal is refused unless the camera was checked and trusted, the count cross-checked and the pushing located in that episode; it must cite that episode's forecast for that zone; code forecasts every action the model skipped and accepts only what the venue-wide rule accepts; and every number in the radio message must match a tool result of the same kind, so a true count cannot be passed off as a density. If the model is unreachable, a deterministic planner calls the same tools under the same rule.
  • Venue-wide choice: each action is judged on the zone and on the waiting area before the entrance it restricts, both priced with the largest error measured in testing. Among actions that keep both under 3.5 people/m², it takes the one that holds the fewest people outside.
  • A record built to be audited: I audit controls for a living, so every decision can be reconstructed. Each trace entry is hash-chained and signed with an Ed25519 key the service can read but not write, and CloudWatch Logs keeps a copy the instance can only append to. The evening report checks both before anything else. Outside the public demonstration, an approval needs a named steward's PIN, with lockout after five wrong tries; the public demo runs in demonstration mode and its decisions are marked "no PIN".
  • Interfaces: the steward console, an evening report rendered from the trace, and a read-only Model Context Protocol endpoint serving the agent's own measurement tools.
  • AWS: one Graviton instance (t4g.small, Amazon Linux 2023), headless OpenCV 5.0.0 aarch64 wheel, the DNN engine chosen per architecture (classic on Graviton, 5.5 times faster there), JuPedSim compiled from source, Caddy with a Let's Encrypt certificate, no SSH (Systems Manager only), an instance role limited to Bedrock, one log group and one bucket, and a hash-chained trace mirrored to CloudWatch Logs.

Results (all reproducible from eval/)

  • The pushing index, from video alone and with no training, separates pushing from walking in trained raters' labels with AUROC 0.798 over 9,260 rated person-seconds (6 Jülich runs, 342 people), and 0.69 to 0.86 within each density band, where a head count carries no signal. The console's own code, at the half resolution it runs at, scores 0.797 on the same rows.
  • At the console's alert threshold, set on three runs and tested on three others, 75 of 78 stretches of pushing were flagged, at the price of flagging 29% of walking person-seconds: every alert goes to a person, never straight to an action.
  • Together with density, the index reaches AUROC 0.831 against 0.786 for density alone (+0.045, 95% interval 0.031 to 0.060, leave one run out): it carries motion information a count does not.
  • Outflow counted from video: 56 of 56 people, error 0.04 people/s.
  • Minutes to danger: warning 3.6 to 4.1 minutes before the zone reaches 4/m², median timing error 32 s (5 simulated evenings).
  • On 7 more venue layouts it was never tuned on: 18 of 18 runs that reached 4/m² warned 3.2 to 5.2 minutes ahead (median error 35 s), and ranking actions on the zone's peak with the first venue's measured error found the best action for the zone in 21 of 21 runs (within 0.05 people/m²).
  • With the console's own alert tiers, over 30 simulated evenings on 10 layouts: all 22 evenings that reached 4.5 people/m² were warned 4.1 to 7.6 minutes ahead.
  • With a second pedestrian model the forecaster was never set on: 19 of 19 evenings that reached 4.5 people/m² were told to act 2.2 to 4.1 minutes ahead.
  • A sudden surge with no trend to read (a quiet street, then the full arrival rate at once): the console told the steward to act 24 to 37 s after it began, and 2.6 to 6.2 minutes before the zone reached 4.5 people/m², in all 23 evenings that got there.
  • Choosing actions venue-wide, over 30 simulated evenings: the second door opened in 30 of 30, the zone stayed at or under 3.01 people/m², a median of 10 people waited outside, and in every evening the choice left the lowest density anywhere of all four actions. Ranked on the zone alone, the same evenings metered both gates and held a median of 912 people outside.
  • Pricing each action's measured forecast error ranks zone peaks correctly: mean regret 0.01 people/m² out of sample, against 1.85 on the forecast alone.
  • Camera watchdog: 84 of 84 rolls, zooms and tilts above 0.25 m caught, 0 false alarms in 132 checks; the ECC screen measures the floor shift to 4 mm in 228 ms on Graviton, and ALIKED + LightGlue (3,046 ms) run only when it sees motion. On the live instance a knock is flagged within 5 s.
  • Counting on oblique street CCTV (CroHD): DM-Count QNRF 10.9% error.
  • Agent: 20 of 20 test episodes reached the right kind of decision with gpt-oss-120b, every proposal one the venue-wide rule accepts. In all 6 pushing episodes it located the pushing (1.4 m before the gate) and said where in the radio message. The code guards sent the model back 4 times, twice for proposing before it had cross-checked the count.
  • Speed on the t4g.small: 528 ms per camera-second for the console's pipeline (718 ms at full size). DM-Count runs in 1.7 s on the classic DNN engine against 9.5 s on the new one; in the live console the agent's second count takes under 6 s.
  • Accessibility: axe-core reports no WCAG 2.1 AA violations on the console (desktop and phone width) or the report.

Pushing index against trained raters, by density band

The pushing index separates pushing from walking within every density band, where a head count carries no signal. AUROC against trained raters' labels, 9,260 person-seconds from 6 Jülich entrance runs, no training.

Seven venue layouts the forecast was never tuned on

Seven street layouts the forecaster was never tuned on, three runs each. Left: every run that reached 4 people/m² was warned 3.2 to 5.2 minutes ahead. Right: regret on the zone's peak when actions are ranked on it, with the error measured on the first venue (blue) and without (grey).

Challenges I ran into

  • Seeking the replay per frame cost 75 ms and could abort FFmpeg; reading forwards cost 3 ms.
  • Measuring outflow by integrating field flux read 40% low; counting matched heads across a line matched all 56 people.
  • The forecast is accurate for actions on the whole inflow and about 2 people/m² optimistic for partial ones (one gate, one extra door). Rather than tune it, every proposal carries the error each action's forecast made in testing, and safety is judged on the largest one.
  • Keeping the zone clear is not the same as keeping the crowd safe: metering the gates empties the zone by holding a median of 912 people outside, where no camera looks. The choice judges every action on the zone and on the waiting area before the gates it restricts, and opens the door instead.
  • A language model follows instructions most of the time, which is not enough for safety: every safeguard the prompt states is also checked in code, and a proposal that skips one goes back to the model as a refusal.
  • ALIKED + LightGlue need about 1.6 GB while they run. On a 2 GB instance, each heavy check runs in a fresh child process, one at a time.
  • On 2 vCPUs and 2 GB, with two models free to run at once, the agent's second count took 75 s and its camera check 23 s. A per-stage benchmark shows OpenCV 5's new DNN engine runs DM-Count 5.5 times slower than the classic engine on Graviton (it is the faster one on x86), so the engine is picked per architecture, DM-Count sees only the zone's bounding box, and an ECC alignment screens the camera before the feature match: under 6 s and 0.3 s.
  • A rotation-plus-shift camera screen reads a 2% zoom as steady. A homography model measures zoom and tilt as well.

Accomplishments I'm proud of

A physics index that agrees with trained human raters without any training, measured in metres and seconds, and an agent whose every number can be traced to a tool result in a tamper-evident record.

What I learned

Driving the real console shows what unit tests miss: a forecast that goes stale before the steward approves, a camera knock that must be caught while a proposal is pending, actions that must be forecast from the same data, and a 2 vCPU instance starved by two models at once. Measuring each stage on the target machine changed a design decision that a desktop benchmark would have hidden. And an objective can be right about the place it measures and wrong about the venue: a safe zone is not a safe crowd.

What's next

Calibrate on a real venue camera (the oblique calibration from floor markings is implemented and unit-tested), validate the pushing threshold on more experiment series, and add upstream cameras so the forecast sees surges before they reach the zone.

Data credits

Forschungszentrum Jülich pedestrian data archive (CC BY 4.0): 2018 entrance experiments, 2022 pushing ratings, 2021 bottleneck experiments. CroHD (CC BY-NC-SA 3.0) for evaluation only.

Built With

Share this project:

Updates

Submission history