Haven: Scene-Aware Shelter Guidance in the Panic Window

Inspiration

In the first few minutes of a sudden disaster, people often panic and choose instinctive shelter that can actually make things worse. Standing near windows during a blast, running into a garage during a tornado, or staying on low ground during a flood are common reactions that increase risk instead of reducing it.

Most emergency apps today provide generic text instructions that assume people already know their environment. We wanted to explore whether a phone's camera could be used to actually "see" the surrounding space and guide users toward safer shelter in real time based on the situation.

What We Built

Haven is a mobile app built with Expo (React Native) that performs a short scan of the user's surroundings and provides scenario-aware shelter guidance.

The user:

Selects a disaster mode — Earthquake, Flood, Tornado, Blast, Fire, or Hazmat
Optionally specifies whether they are indoors or outdoors
Performs a 360° scan of their environment

The result is displayed on a single photo from the scan with overlaid safe zones, danger zones, and exit paths. These overlays are drawn using normalized bounding boxes and paired with a short voice summary explaining the safest immediate action.

How It Works

Haven captures frames either using the device gyroscope (one frame every 15°) or a time-based fallback when sensors are unavailable. A small set of key frames is selected and sent along with a disaster-specific prompt encoding rules such as:

Earthquake: Drop and cover under sturdy furniture
Flood: Move to higher stable surfaces
Tornado: Seek lowest interior shelter
Blast: Stay below the window line

The vision model returns structured JSON containing safety zones in normalized $ [0, 1] $ coordinates, an overall safety score, and a recommended action. Zones are validated, clamped if necessary, and rendered directly onto the result photo.

The bounding box coordinates follow this constraint:

$$x_1, y_1, x_2, y_2 \in [0, 1]$$

where $ (x_1, y_1) $ is the top-left corner and $ (x_2, y_2) $ is the bottom-right corner, both normalized to the anchor frame's width and height.

Challenges We Faced

Overlay alignment was a major issue early on. We solved this by selecting a canonical result frame and explicitly requiring all model-generated bounding boxes to be relative to that frame's coordinate space.

Inconsistent gyroscope support across devices led us to implement a time-based capture fallback to maintain reliability.

Noisy model output was the third challenge — vision models often wrap outputs in extra text or markdown. We added robust parsing and validation to safely extract structured zone data:

{
  "bestRefuge": { "label": "GET HERE", "bbox": [0.1, 0.4, 0.5, 0.9], "confidence": 0.82 },
  "dangerZones": [{ "label": "AVOID", "bbox": [0.6, 0.0, 1.0, 0.4], "confidence": 0.91 }]
}

What We Learned

Disaster response is highly scenario-dependent. The same object may be safe in one situation and dangerous in another — which meant encoding disaster-specific logic into prompts rather than relying on a single generic rule set.

We also found that directly requesting spatial safety zones from the model avoided many of the failure modes of traditional detect-then-classify pipelines.

Why It Matters

In the panic window, a single clear indication of the safest nearby refuge — along with visible hazards on the actual room photo — can help users avoid instinctive but unsafe decisions.

Emergency preparedness knowledge is unequally distributed. Haven democratizes spatial situational awareness in a crisis, making guidance specific to the environment rather than generic instructions that assume prior knowledge.

How It's Different

Most emergency preparedness apps provide static checklists or text-only advice. Haven uses a multi-frame scan and vision-based reasoning to interpret the user's immediate surroundings and overlay safe and unsafe regions directly onto the captured scene — making guidance specific to the actual environment in front of you.