Inspiration
When someone collapses, help is often only a few meters away: a defibrillator on a wall, a stranger who knows CPR. What is missing is knowing where it is, and getting the right person there fast. Building owners rarely know how far their AEDs, extinguishers and exits really are from every spot inside, and 911 rarely knows about AEDs nobody registered.
We wanted one map that does both jobs: measure how far every spot in a building is from the things that keep you alive, and use that same map in the first minutes of an emergency.
What it does
The census. For every scan point in a building's Matterport twin, Within Reach computes the walking time to the nearest AED, fire extinguisher and exit, flags the red zones, and finds where a single new AED would help most. A LangChain Deep Agent plans the work (Records, Scanner, Mapper, Outside and Fixer subagents) and streams every decision to an on-screen console. The Fixer drafts an owner letter, a 911 registry list and a one-page emergency card. We always say "no AED visible in the scan", never "no AED".
The live response. A laptop webcam plays a building's security camera.
- When someone goes down, the speaker asks "Are you okay? If you can hear me, raise your hand."
- With no raised hand after 5 seconds, it is treated as a possible cardiac arrest. The speaker coaches hands-only CPR and says where the AED is, using the census map: "get the AED, 14 meters, near the Gothic arcade gate."
- The nearest responders get an alert in the app and by iMessage. The first to accept becomes the AED runner.
- The runner's phone shows the floor plan, and "View route in 3D" opens the Matterport twin, which walks them to the AED and back at walking pace.
- A simulated 911 dispatcher gets a structured briefing: what happened, where, how long ago, CPR and AED status, and the callback number.
- The dashboard shows the timeline and a Utstein-style handoff card.
Our live demo building is Princeton's Julis Romo Rabinowitz Building. No AED is visible in its scan, so we placed a clearly labeled demo AED and kept it out of every census number.
How we built it
- Data: the Matterport Model API gives us every sweep's position and its panoramas. Its neighbor links are line of sight, so some cross floors straight over the open atrium. We rebuilt a walking graph that keeps short same-floor edges and joins floors only through stair sweeps.
- The math is plain Python and networkx, not the LLM. Walking time from scan point $s$ to equipment type $k$ is a multi-source Dijkstra over the walking graph at $v = 1.4$ m/s:
$$T_k(s) = \min_{a \in A_k} \frac{d(s, a)}{v}$$
A spot is red for AEDs when the round trip $2\,T_{\text{AED}}(s)$ exceeds 180 s, and for extinguishers and exits when the one-way distance passes 22.9 m and 61 m (configurable approximations of the guidelines, labeled as such). Buildings are ranked by a gap score
$$G = 0.5\, r_{\text{AED}} + 0.25\, r_{\text{ext}} + 0.25\, r_{\text{exit}}$$
where $r$ is the red fraction of reachable floor. The Fixer places one new AED at
$$s^* = \arg\min_{c} \; \max_{s} \; T'_{\text{AED}}(s \mid c)$$
over candidate sweeps $c$.
- Live path: FastAPI and a deterministic LangGraph state machine, with YOLOv8-pose for fall, raised-hand and seizure-like motion detection. No LLM sits before the first spoken instruction; the only model call on that path, a scene check, can add safety steps but never remove or delay one.
- Alerts: Supabase Realtime for the app, Photon for iMessage, and Twilio wired for phone calls.
- Apps: a Next.js dashboard (city map, building heatmaps, the "drop an AED" animation, the live desk) and an Expo responder app.
- The 3D walk: the app streams its walking dot to the 3D page over a Realtime broadcast channel. Each move between scan points flies for as long as walking it would take, facing the direction of travel, with heading $\psi = \operatorname{atan2}(-\Delta x, -\Delta z)$, a convention we measured on the model.
Challenges we ran into
- Indoor position is hard. Phone GPS indoors is off by meters. We georeferenced the scan from four measured spots, which gave a fit error of about 3.6 m. On top of that, the app moves the responder by step count. For a reliable demo we added "Walk it", which replays the route at walking pace.
- Two coordinate systems. Matterport's Model API is z-up and its viewer is y-up, so we convert with $(x, y, z) \mapsto (x, z, -y)$. We verified that all 66 sweep IDs match between the API and the 3D viewer before trusting a single route.
- Getting the 3D view onto phones. The viewer key only worked on approved domains, so the phone page failed until we got our deployed domain added. The campus Wi-Fi also dropped connections to our site in bursts.
- Triage is unforgiving. A raised arm during one test fall made the system read a conscious person, so it correctly alerted only staff. That taught us how much the clear, deterministic signals matter.
- Messaging rules. Shared iMessage lines can only reply in chats the person started, so every demo phone had to text the line once.
Accomplishments that we're proud of
- In our test runs, responder alerts went out 5 to 6 seconds after the camera reported a collapse (the voice check takes 5 of those), and a responder accepted 4 to 8 seconds later.
- The 18 m route to the demo AED plays in 3D in about 13 seconds at walking pace, and the view switches to the way back as soon as the runner picks up the AED.
- 460 Python tests and 66 dashboard tests, a strict "real numbers only" rule, and a live path that never depends on an LLM.
What we learned
Be agentic where it is safe, and deterministic where lives are at stake. Let the agent plan and explain the census, but keep the first minutes of an emergency in a state machine you can test. And be honest about data: what a scan shows is not the whole truth about a building.
What's next for Within Reach
- City scale: run the census across CoStar's buildings once we have dataset access. We did not get it this weekend, so there is no city-wide headline number yet.
- Real calls: turn on Twilio for responder calls, and send 911 a data feed through a monitoring center instead of a voice call.
- Better indoor position: beacons or ultra-wideband (UWB) instead of GPS.
- For owners: put a real AED trainer in a fresh scan and let the scanner find it on its own, and give owners the fix list and emergency card for their buildings.
Built With
- cloudflare
- expo.io
- fastapi
- langchain
- langgraph
- leaflet.js
- matterport
- networkx
- next.js
- numpy
- opencv
- photon
- postgresql
- pytest
- python
- react
- react-native
- supabase
- tailwindcss
- twilio
- typescript
- uv
- vercel
- vitest
- yolov8