RadAR

Save lives by using Lidar and Computer Vision to build accurate and real time 3-D world models for emergency responders.

Comment

Inspiration

First responders run into buildings and rooms they've never seen to save lives, usually with almost no picture of what the people next to them are seeing. SWAT teams and firefighters especially have to commit before they know the layout, the exits, or where their own team actually is. Command sits outside, piecing the scene together from radio calls. RadAR started from that: give everyone on the call live spatial awareness, so the whole team and their commander are looking at the same world instead of shouting over the radio.

What it does

RadAR turns iPhones and iPads into live body cams that build a shared 3D map of a building as responders walk it, and puts that map in front of a commander in a browser.

  • A live 3D map shared by the team. Every device streams 30 fps video, its ARKit position and orientation, and LiDAR depth on Pro devices. The server fuses all of it into one map that grows in real time. When one responder rounds a corner, that geometry shows up on the map for everyone.
  • Mission-control hub. Any browser over HTTPS shows the 3D map, live video from every responder, each responder's position, and the path they've walked drawn on the floor in their colour. You can fly through the map or view it from above. The hub can show surfaces as photos draped over the LiDAR depth, or as points.
  • Where your teammates are. Each responder's video feed, both on the hub and in the app, carries an arrow and a distance to every other responder. Distance comes from ultra-wideband (Nearby Interaction) between the phones, and direction from the camera positions.
  • Vitals and head tracking from AirPods. Live heart rate (through HealthKit) appears on each responder's feed. AirPods head tracking shows where each responder is actually looking, as a field-of-view outline on the map.
  • Push-to-talk radio with live transcripts. Responders and command talk over a built-in voice channel. AWS Transcribe turns every call into a timestamped transcript on the hub, so nothing said on the radio is lost.
  • Record, replay, enhance. Command can record a session under a name. Everything is saved and streamed to S3 as it happens: video, positions, the map, walked paths, vitals and transcripts. Past maps load instantly or replay on their original timeline with video. An offline pass can re-texture a saved map with sharper photos from the full-resolution video.

How we built it

  • iOS app (Swift, SwiftUI):
    • ARKit world tracking with LiDAR scene depth, and 1920×1440 H.264 hardware video encoding at 30 fps.
    • Nearby Interaction for UWB distance between phones, HealthKit for AirPods heart rate, CoreMotion for AirPods head tracking, and an AVAudioEngine push-to-talk call.
    • Phones join by scanning a QR code on the hub (VisionKit); nobody types an IP address.
  • Python relay (asyncio WebSockets):
    • Every device and every hub connects to one relay. Video, positions, depth, audio and vitals share one session and one compact binary protocol.
    • For LiDAR devices, the relay sends the hub a textured "patch" (the photo draped over its depth) every 25 cm or 15° of camera movement.
    • The relay writes the session file and streams it to S3 in parts.
  • Fine-tuned 3D foundation model + our metric calibration layer: We started from lingbot-map and fine-tuned it for our use case, then run it on an NVIDIA L40S to turn live camera frames into depth (~52 ms per frame). Even after fine-tuning, its output has no real-world scale and its own coordinate frame, so we built a math layer on top of it that adapts the model to each phone in real time:
    • Sim(3) alignment to ARKit: the model and ARKit track the same camera every frame. We solve for the rotation, scale and translation that map the model's frame onto ARKit's metric world. Rotation is averaged from camera orientations through an SVD projection onto SO(3), so it holds even when walking a straight hallway.
    • Per-frame metric scale: each frame's depth is rescaled by the median ratio between ARKit's measured feature points and the model's predicted depth at the same pixels.
    • Confidence and range filtering: low-confidence and distant predictions are dropped before they reach the map.
    • Multi-phone deduplication: points are merged into one shared voxel grid, so two responders mapping the same hallway build one map, not two.
    • LiDAR fusion: on Pro devices, ARKit's metric LiDAR depth is upsampled to the camera image with an edge-aware filter and replaces the model's estimate entirely.
    • The three models load at startup, so two phones can map at once.
  • Hub (three.js): a browser 3D view of the combined map plus live video, vitals, direction arrows, head-tracking outlines, the transcript, and recording and replay controls.
  • AWS:
    • The relay, hub and GPU worker run on one EC2 g6e.xlarge (L40S) in us-east-2, behind Caddy with an automatic Let's Encrypt certificate. Phones on cellular connect from anywhere.
    • Sessions live in an encrypted, private S3 bucket, and AWS Transcribe streaming handles the radio transcripts.
    • Enhancement jobs run on a separate EC2 box that the relay starts on demand and that shuts itself down after three idle minutes.
    • Instance roles handle access (no stored keys), and admin is through Systems Manager (no SSH).
  • Map enhancement: a CPU job decodes the recorded video and re-textures each LiDAR patch from the sharpest nearby frame. It only uses frames that still see the whole patch, and places each photo with that frame's exact camera position. A 189-patch map enhances in about 25 seconds.

Challenges we ran into

  • Making a foundation model metric and shared. Depth from video alone has no scale, drifts, and two phones produce two different worlds. Fine-tuning the model for our setting helped, but it wasn't enough on its own, so we also built a geometric calibration layer (Sim(3) alignment to ARKit, per-frame scale from ARKit feature points, confidence filtering, shared voxel merging) that puts every phone's output into one metric coordinate frame live.
  • Live, with several devices at once. The GPU path is slower than 30 fps video. We decoupled the two so video is always live and the map trails slightly behind, shown on the hub as "map lag". We also fixed GPU queue flooding and model-loading races that let only one of two phones map.
  • Bandwidth. Each phone sends about 3 Mbit/s of video plus depth. We had to tune depth resolution, how often patches are sent, and point removal so it works over a hotspot or cellular.
  • Phone-to-phone direction and distance. Getting UWB distance between phones reliably meant retries, falling back when camera assistance isn't available, and combining UWB distance with the direction from ARKit.
  • Sharper maps without breaking them. Our first enhanced view smeared and streaked. We rebuilt it to keep the exact geometry and only swap in sharper photos, rejecting frames that don't fully see a surface.

Accomplishments that we're proud of

  • The full loop runs live on AWS. Two responders walk a building and command watches it appear in 3D, with their video, heart rate, the direction and distance between them, where they're looking, and a transcript of every radio call.
  • Everyone works from one shared, metric map, not separate camera feeds.
  • Every session is recorded and replayable, which makes after-action reviews and training possible right away.
  • It runs on hardware responders already carry: an iPhone and AirPods.

What we learned

How to combine LiDAR and video into live 3D maps, and how little of the work is the model itself. Most of it is positioning, scale, fusing devices, and keeping several of them in one coordinate frame. A fine-tuned foundation model gets you most of the way, but making its output usable in the real world (metric, consistent over time, and shared across devices) took as much work as running the model itself. We also learned how much the surrounding features matter to the people using it: where teammates are, their vitals, and the radio.

What's next for RadAR

  • Accountability for downed responders. Automatic alerts when a responder's heart rate spikes or they stop moving, and one-tap guidance to their last known position.
  • Room-by-room clearing status on the shared map for SWAT and police teams.
  • 3D maps of buildings made in advance (schools, hospitals, high-rises), ready for crews before they arrive.
  • Seeing through smoke and dark with thermal sensors, and support for body cams beyond the iPhone.
  • More responders per scene, with tighter maps and a command view that's useful before the team clears the first room.

Built With

Share this project:

Updates

Submission history