smoke-rescue

Quadruped robots search a burning house with no floor plan, joining their maps over a lossy radio with OpenCV 5 visual SLAM, so they can find survivors before firefighters go in.

Comment 2

Smoke Rescue

Quadruped robots search a burning house with no floor plan, joining their maps over a lossy radio with OpenCV 5 visual SLAM, so they can find survivors before firefighters go in.

Inspiration

My uncle, a retired firefighter, has regaled me with his stories of how he handled burning buildings. One point stuck with me though - a story where his colleague and a resident were caught under the rubble because he was unable to find them in time. Technology has advanced towards solving this, and we wanted to build something that contributes to this. In a house fire, smoke takes away what firefighters can see, and heat takes away their time. Furthermore, if they can't share what they've seen, they may end up searching the same rooms twice.

  • ~180,000 deaths a year worldwide are caused by burns, about one every three minutes (WHO)
  • 95%+ of fatal fire-related burns happen in low- and middle-income countries (WHO EMRO)
  • 3,920 civilians died in US fires in 2024, up 6.8% on 2023 (NFPA)
  • ~3 in 4 of those deaths were in homes and apartment buildings (NFPA)

WHO totals cover all burns, not only fires.

Charleston, June 2007. Nine firefighters died in the Sofa Super Store fire in South Carolina, a furniture showroom and warehouse. They became disoriented inside and ran out of air. The crews inside could not see the fire that was spreading above them, and the first radio call for help was not heard on the fireground. Investigators cited many failures, from command and radios to the lack of sprinklers. But at its heart it was the problem this project is about: people inside a smoke-filled building, with no clear picture of where anyone was or what was happening around them. (NIOSH summary, timeline)

Hong Kong, November 2025. Fire tore through seven of the eight towers at Wang Fuk Court. 168 people died, and at the worst point hundreds were reported missing. For days the question was the one our robots are built to answer: which flats have been checked, and who is still inside? (final toll, search coverage) We haven't tested on a tower, so we won't claim we'd have changed that outcome. The job we're building toward is the same, though: go in first, mark every room as searched or not, and report back while there's still time.

Robots get lost too, in a quieter way. Their legs slip, so the map a robot is drawing slowly goes wrong the longer it walks. Our fix is loop closure: when a robot recognises a place it has already seen, it checks its position against it. We kept everything else hard too. Each robot sees only through its own RGB-D camera, thermal camera and LiDAR, nobody gets a floor plan, the radio is lossy, and every result is scored against ground truth no robot code can read. The question we kept coming back to: how little can a robot team be told, and still work as a team?

What it does

  • A team of quadruped robots starts at the front door of a burning, smoke-filled house and searches it for trapped people. None of them knows the layout.
  • Each robot maps as it walks. It builds its own map from LiDAR and depth and keeps its heading steady using the gyro plus LiDAR scan matching.
  • They check their own position. Comparing what a robot's camera sees now with places it saw at least 20 seconds earlier shows how far its map has slipped. The map is adjusted only if the LiDAR still fits it at least as well afterwards.
  • They find each other without a shared map. ORB feature matches between camera views tell each robot where a teammate is, and LiDAR double-checks it.
  • They divide the house. They radio which rooms are searched and which doorways are taken, so they don't repeat each other's work.
  • They work around fire. They back away from rising heat, keep a way out, and spray water (12 L tank) on fire they measured with the thermal camera. When there's nothing left to search, they walk out.
  • A person only counts when a camera sees them. A person is confirmed only when a YOLO11m detection, run through OpenCV's cv2.dnn on the robot's own colour frame, fires. A warm shape alone never counts.
  • An LLM agent on each robot, plus a commander, decides where to search and whether a warm shape is a person. The agents work through OpenCV 5 tools. Each tool result carries an evidence id, and the agent must cite it to act. Two checks stand between the agent and any action: a Python guardrail verifies the evidence, then the robot's own brain checks the move against its map and heat readings. If either refuses, the reason goes back to the agent.

Agentic Vision

Each robot runs a GPT agent in a perceive-decide-act loop over OpenCV 5 tools (YOLO11m via cv2.dnn, thermal contours, depth-based flat-surface check). Every OpenCV result gets an evidence id; risky actions (confirm survivor, spray, open door, extract) must cite one, and code refuses them otherwise, sending the reason back to the agent. Trace example (seed 934, robot B): inspect_candidate -> uncertain (person detection clipped at the image edge) -> agent turns to face it -> inspect_candidate -> person (YOLO11m confidence 0.92 at 2.62 m) -> confirm_survivor citing that id -> radio report. In that mission the two robots' agents gave 127 orders: code passed 94 and refused 33. Across 34 recorded runs, 990 robot orders cite OpenCV evidence and 169 were refused. Every model reply, tool call, refusal and latency is logged (agent_trace.jsonl). Agents can request human approval citing evidence (with APPROVAL_MODE=human the request stays open until a person answers it). Evidence: report/agentic_vision_evidence.pdf, diagrams/agent_workflow.svg, run_logs/unity_house_onedoor_seed934/agent_trace.jsonl.

How we built it

  • The world: Unity 6 HDRP simulates the house, fire, smoke and sensors. Sensors are seeded and deterministic: RGB-D at 15 Hz with a depth noise model, a thermal camera, 720-beam LiDAR where smoke drops pulses, an IMU and leg odometry.
  • The robot brain: C#, one per robot, running at 60 Hz. It handles mapping, frontier exploration, A* path planning with fire and wall clearance, stuck recovery and the heat reflexes. Every threshold the robot uses is in config/brain.json, with the reason or measurement behind it.
  • Perception: a Python 3.11 FastAPI service running OpenCV 5.0. It does person detection (YOLO11m ONNX in cv2.dnn), place recognition for visual SLAM (matching ORB features between camera views, then fitting the rigid motion between them with RANSAC), a live per-robot 3D map (cv2.Volume HashTSDF), and fire and heat analysis (thermal thresholding and contours).
  • Linking the robots' maps: each robot broadcasts ORB keyframes (7.6 KB each, under the radio's 8 KB limit). A match needs at least 10 features agreeing on one motion, and is then cross-checked by laying the teammate's LiDAR sweep over the robot's own map. Two agreeing matches taken at least 1 m apart refine how the maps line up.
  • Drift check: the same ORB matcher runs against the robot's own older keyframes. In our best full mission it found 138 matches. Robot A's median drift was 3.1 cm and 0.25°, so none needed a correction. In simulation the drift stayed small, so this worked as a check rather than a repair.
  • Scoring: the evaluator reads ground truth from a separate Unity assembly, and a pytest check fails if robot code can import it.
  • Cloud (AWS): CloudFormation in us-east-1. The agent and OpenCV API run on ECS Fargate (ARM64) behind an ALB with a WAF rate limit, plus a Batch job for offline SLAM, budget alarms, and a results site on a Lambda Function URL.

The maths behind it

Three quick calculations shaped how we built it:

  • Why a tiny heading error matters. A robot that starts 2.39° off its pad heading (as robot B did in our main test house) misplaces a wall 15 m away by \( 15\,\text{m} \times \tan(2.39^\circ) \approx 0.63\,\text{m} \). That is enough to put a doorway inside a wall, so the robots cannot just trust their launch pads. They correct the link with camera and LiDAR matches.
  • Why water runs out fast. In our simulation, a 12 L tank absorbs at most \( 12\,\text{L} \times 2590\,\text{kJ/L} \times 0.5 \approx 15.5\,\text{MJ} \) of heat. At 1 L/s the tank is empty after 12 seconds. So the robots spray only fire they have measured, and only when it blocks the way or the search.
  • Why we quote intervals, not just percentages. A 94.9% success rate over 39 missions has a 95% interval of 87.2% to 100%. A single percentage would hide that.

What we learned

  • One good run proves nothing. A single rendered mission can look perfect, so we report intervals across 30 generated houses and 39 real layouts as well.
  • The recordings found bugs our tests didn't. Until 25 Oct, no live run ended with both robots outside. Three causes turned up, each fixed with a regression test that fails without the fix. The robot locked the floor at its way in, so the walk-out had no route. A robot left while the team still had unentered rooms. And the first robot to finish parked in the front doorway and blocked the only door.
  • Two promising ideas didn't survive measurement. OpenCV's own rigid fit with uniqueness and spread gates didn't beat our place matcher on held-out TUM data (recall 74.4% vs 75.1%, 43 vs 46 wrong poses), so we kept ours. cv2.Odometry as the pose source drifted 17° in 4 s on our frames, so we dropped it.

What's next

  • Missions where the robots enter by different doors, which is where linking is hardest.
  • Better cross-robot place recognition, so robots link up faster and more reliably.
  • A harder benchmark: a fire head start before entry, a 300 s limit, and people reachable only through another room. Our current tests aren't challenging enough to show a meaningful difference in how many survivors each approach finds.
  • Support for non-rectangular rooms and wide openings, so more real floor plans qualify.
  • Testing the perception stack on real robot data, not just simulated and TUM frames.

Realism factors considered

Apart from the constraints already explained above, some other realism factors we considered are:

  • An imperfect starting point: the robots launch from pads at the front door, which gives them a rough shared frame, but real pads are never exact. A robot also counts floor as indoors only from its own evidence.
  • Imperfect sensors: depth noise, LiDAR pulses lost in smoke, leg-odometry slip, and a thermal camera that has to tell people from warm appliances.
  • An unreliable radio: lossy, short-range, with 8 KB payloads. Anything a teammate reports is converted using the offset our robots measured between their maps, never the true one.
  • Fire and water: heat-driven fire, rooms marked burning from hot air, and a published water-absorption model (Grimwood 2005) in which wetted fuel resists reignition. The houses in our demo video use a slow t-squared fire growth curve.
  • Beyond our own houses: 39 real floor plans from HouseExpo, and place recognition and the person detector checked on real TUM RGB-D data.

Results / evaluation

All metrics are computed offline against ground truth the robots cannot read. The baseline is frontier-only exploration; the final system is the full two-robot team. The held-out set is 39 real HouseExpo floor plans, run with the full system only. Values are means with 95% confidence intervals.

Survivor detection

  • Recall: 96.0% (92.0-98.7), and 99.0% (97.4-100.0) on the held-out layouts.
  • Success rate (every survivor found, zero false reports): 83.3% (70.0-96.7), and 94.9% (87.2-100.0) on the held-out layouts.
  • Found by 240 s: 94.0% against 72.0% for the baseline.
  • Time to first detection: 26.0 s against 40.3 s for the baseline.
  • False positives: 0.20 per mission (0.00-0.53), and 0.00 on the held-out layouts.

Team versus one robot and no radio

  • Against no radio (30 houses): people confirmed 37 s sooner (p below 0.001) and 2.8 points more floor covered (p = 0.0005).
  • Against one robot (30 houses): people confirmed 60 s sooner (p below 0.001) and 5.3 points more floor covered (p below 0.001).
  • Real layouts (39 HouseExpo floor plans): two robots found 99.0% of people against 92.3% for one robot, a gain of 6.7 points (95% interval +3.1 to +11.3, p = 0.002), with no false confirmations in either variant.
  • The final share of people found was about the same for every variant (94-98%) on the generated houses. The team's edge there was speed, not the final count.

Exploration and coverage

  • Floor covered: 95.3% (93.7-96.7) against 90.1% for the baseline, and 96.9% on the held-out layouts.
  • Rooms under 50% coverage: 0.27 per mission against 0.60 for the baseline.

Map quality

  • Map recall: 99.7% (99.2-100.0) of the observed walls are mapped.
  • Map precision: 43.9% (41.0-47.0). This is scored against walls only, so furniture counts against it.

Localization and map merging

  • Position error, median: 0.092 m (0.082-0.103) on the held-out layouts.
  • Heading error, median: 0.22° (0.20-0.24) on the held-out layouts.
  • Map merge in the rendered family ranch layout: the two robots' maps linked at 44.3 s, with 0.036 m and 0.54° of error and no false merges.

Robot safety in fire

  • Heat exposure: 113.7 s (73.6-164.7) above 60 °C, against 50.8 s for the baseline.
  • Collisions: 0.00 s in the rendered family ranch layout.

Rendered Unity layout

  • Family ranch: 5 of 5 survivors found, 0 false confirmations, both robots walked out.

Agent behaviour

  • Across 34 recorded runs, the agents made 2,727 OpenCV tool calls, and the evidence guardrail refused 169 orders that lacked supporting evidence.

The full metric tables are in the evaluation document.

How to use

  • Run it yourself:
cd brain && uv sync && uv run pytest -q
uv run uvicorn brain.agent.server:app --port 8080
./tools/headless/test.sh   # C# brain logic, no Unity needed
  • Live Unity run: Unity 6000.6.6f1, scene House.unity, then the menu Tools, Smoke Rescue, Record, House one-door seed 934 full mission.
  • Score a recording against hidden ground truth: python tools/run_audit.py data/recordings/RUN_NAME
  • Cloud: deploy/aws/deploy.sh --email YOUR_EMAIL ..., then deploy/aws/smoke_test.sh YOUR_URL --direct

Honest limitations

  • It's a simulation. The fire model is an approximation, not a validated fire-engineering solver, and we make no real-world rescue claim.
  • Person detection from a single camera view often misses people, so robots have to get closer before they can confirm someone.
  • Cross-robot linking works because many keyframes are sent and only two agreeing matches are needed. Robots that never share a view won't link.
  • The 39 real layouts are the boxy, rectangular-room subset of HouseExpo (39 of 17,084 candidates), with open-plan openings narrowed to 1 m doors. We have no like-for-like comparison with published exploration numbers.
  • Our houses are single-floor with rectangular rooms. Multi-storey towers like Wang Fuk Court, with stairs and many more rooms, are beyond what we have tested.
  • Recording is slow, about 5 simulated seconds per wall-clock minute with live perception, so a full mission takes about two hours.

Built with

  • Unity 6 (HDRP), C#, Python 3.11
  • OpenCV 5.0 (cv2.dnn, ORB, BFMatcher, HashTSDF, contours), YOLO11m (ONNX)
  • FastAPI, Uvicorn, NumPy, SciPy
  • GPT via Azure OpenAI (agents)
  • AWS: ECS Fargate (ARM64), Application Load Balancer, WAF, Lambda, S3, ECR, Batch, Secrets Manager, CloudWatch, CloudFormation
  • Docker, RunPod (benchmarks), pytest, uv
  • ORB-SLAM3 (offline SLAM comparison), Qwen2-VL (decision narrator)
  • HouseExpo an `d TUM RGB-D (evaluation data)

Built With

Share this project:

Updates

Submission history