SafeSpot

Inspiration

When a tornado warning goes off, your phone tells you that danger is coming. It doesn't tell you where to go. The standard advice is "get to an interior room on the lowest floor, away from windows," but in a panic, most people don't know which room in their own home that is, let alone what to do if the hallway to it is blocked.

We wanted to close that gap. Safe Spot turns a real 3D scan of your home into guidance to the safest spot, and reroutes the moment part of the house is damaged.

What it does

Open the app and tap Tornado, Earthquake or Fire.
Safe Spot works out which room you're in from the camera, motion sensors, the floor plan and the home's own scan photos.
It picks the safest target for that hazard and guides you there in first person, through a photoreal 360° walkthrough of your actual home, with an arrow toward the safe spot, a minimap and spoken guidance.
If a room is reported damaged, the backend recomputes the route and pushes it to the phone instantly.
In an earthquake, it tells you to take cover immediately, listens to your phone's motion, and only sends you to the exit once the shaking has stopped.

How we built it

Data. We used CoStar Group's Matterport data for a real home in Houston. The Matterport Model API (GraphQL) gave us rooms and room types, wall outlines, doorways, windows, 19 panorama scan positions and a model-to-latitude/longitude transform. We wrote a script that introspects the API schema and pulls everything about any scan into one JSON file, and used the scan's Matterpak bundle (a full-detail textured 3D model) for the photoreal house.

Safety planning. Rules are code, not guesses. For a tornado, the planner prefers rooms on the lowest floor with no windows and no outside walls (interior closets, bathrooms, hallways), nearest first. Earthquakes mean take cover now, exit after the shaking stops. Fires mean the nearest exit whose path avoids the fire and blocked rooms, with a second exit as backup.

Routing. Rooms and doorways form a graph, and we find the shortest safe path with A*:

$$f(n) = g(n) + h(n)$$

where $g(n)$ is the distance walked so far (summed between room centres) and $h(n)$ is the straight-line distance to the target. Blocked rooms are removed from the graph, and the route is recomputed and pushed to the phone over a WebSocket on every change.

Indoor positioning. GPS can't tell rooms apart, so the iPhone uses ARKit camera tracking, aligned to the floor plan with two taps. Every movement is snapped to the plan: you can't pass through a wall, and you only change rooms through a doorway. The camera also compares what it sees with 228 views cut from the home's 19 panorama scans (Apple Vision image feature prints) to recognise the room. If the camera is lost (dark, smoke, phone in a pocket), it switches automatically to step counting (0.74 m per step, measured) plus gyroscope turns; we measured the compass drifting 15–90° indoors, so it isn't used for turns. After about 10 m without the camera, it asks the user to tap where they are. GPS only confirms you're at the right building.

Earthquake detection. During earthquake mode the phone streams a once-per-second summary to the backend: the RMS of acceleration over the last second,

$$a_{\text{rms}} = \sqrt{\frac{1}{N}\sum_{i=1}^{N} \lVert \mathbf{a}_i \rVert^2}$$

(in g, gravity removed), plus Apple's activity label. The backend's state machine declares shaking above 0.30 g and calm below 0.08 g, holds take-cover for at least 12 s, and needs 7 s of calm before routing to the exit. A strong reading afterwards counts as an aftershock and sends you back under cover.

Agents. A LangChain Deep Agent turns messy NWS tornado warnings and USGS earthquake alerts into room states, using tools that expose only real rooms from the house data. Every output is validated against the house data, and if the model fails, the backend falls back to a deterministic risk assessment, so the route never depends on the model getting it right. One-tap reports ("blocked here") go straight through without a model.

The app. A native iOS app in Swift and SwiftUI, with RealityKit for the 3D house, photoreal 360° panoramas, a FastAPI backend, and spoken guidance with AVSpeech.

Challenges we ran into

No furniture in the data. The scan has rooms, walls, doors and windows but no tables or tubs (Matterport's object detection isn't available for it), so "take cover under the table" isn't possible yet. The planner falls back from safe room to exit; detecting furniture from the scan photos is next.
Indoor positioning is hard. GPS doesn't work inside, the compass is unreliable near steel and wiring, and the camera can fail in exactly the conditions that matter, so we built the camera → steps + gyro → tap fallback chain.
Recognising rooms without fooling ourselves. The rooms look alike (white walls, the same carpet). Single-view matching picked the right room only 38% of the time; comparing just the current room and its neighbours across 4 views raised that to 83%. We also tested against a different apartment's scans to make sure the app never "recognises" somewhere that isn't the house: 0 of 864 false matches with our final thresholds.
Demoing a Houston house in Princeton. We overlaid the scanned floor plan onto an open space at the venue and lined it up with two taps.
Messy data and plumbing. Shared walls appear once per room, some spaces are zero-height "voids", asset links expire within hours, the full 3D model was hidden under "bundles" rather than "resources", and getting a phone to talk to a laptop backend on venue Wi-Fi took local-network permissions.

Accomplishments that we're proud of

Real windows, outside walls and doorways from a real scan drive the safety decision.
Live rerouting around a blocked room, pushed to the phone instantly.
Positioning that degrades gracefully: camera, then steps and gyro, then a tap, without the dot ever jumping or walking through a wall (tested against all 8 interior doorways and 40 wall crossings in the house).
An earthquake flow that never tells you to run while the ground is shaking, driven by the phone's real motion.
An agent that can't hallucinate a room that doesn't exist.

What we learned

How much spatial data a 3D scan holds: room classifications, wall outlines, opening types (window, doorway, opening, divider), panorama positions and a transform from model coordinates to latitude/longitude.
How to combine an AI agent with deterministic code: let the agent judge, let code do the math, and validate everything the agent says.
How indoor sensors behave in practice: step counting is reliable (our two distance estimates agreed within 2%), the compass isn't, and every threshold should be tested against cases that should fail, not just ones that should pass.

What's next for Safe Spot

Detect furniture (tables, tubs) from the scan photos so Safe Spot can say "get under the kitchen table".
Every building with a scan.
Building managers who can send shelter guidance to every tenant at once.
Accessibility routes for wheelchair users and people with limited mobility.
A responder view showing which rooms people sheltered in and who needs help.
More hazards: hurricanes, floods and wildfires.