Iris

Inspiration

After an earthquake or flood, the first 48 hours decide most outcomes. Rescue teams have radar, acoustic sensors, and dogs for finding people deep in debris, but the first sweep of a site still comes down to responders scanning by eye, tired and moving fast. A hand sticking out of rubble or a person half underwater is easy to miss. We wanted a second set of eyes that costs almost nothing, needs no extra gear, and never takes the responder's hands or attention off the scene.

What it does

Iris runs on Meta smart glasses. As a responder walks a site, the glasses stream what they see to an iPhone, which sends frames to our server. A detection model looks for people in rubble, storm, and flood conditions, and when it's confident, Iris alerts the responder two ways: a bounding box appears around the person on the glasses display, and the glasses speak a short cue like "possible person, left." The responder sees exactly where to look without pulling out a phone or taking their hands off the scene.

Iris is not meant to see through rubble. It's a surface-search aid that helps responders catch partially visible people faster, and it's meant to complement the tools teams already use.

How we built it

App: An iOS app built on Meta's Wearables Device Access Toolkit pulls frames from the glasses, downsizes them, and posts them to the server a few times per second. It draws the returned bounding boxes on the glasses display and speaks alerts with text-to-speech.
Server: A FastAPI server runs detection on each frame and decides when to alert.
Model: We fine-tuned YOLO26 on rubble and flood datasets from Roboflow, merged into one person class. We added weather augmentations (rain, fog, motion blur, low light) so it holds up in the conditions it would actually be used in.
Alert logic: A single detection never triggers an alert. Iris only alerts when a person shows up in at least 2 of the last 3 frames, which cuts false alarms without adding much delay.

We tuned for recall over precision. A false alarm costs a responder a glance. A miss could cost a life.

Challenges

Viewpoint gap. Most disaster datasets are shot from drones, but glasses see from eye level. We combined aerial and ground-level data and tiled the aerial images so small figures weren't lost.
Messy labels. Every dataset labeled people differently, so we wrote a merge script to remap classes and filter out bad sources.
Limited GPU compute. We trained in Google Colab and kept running out of GPU compute, which forced us to train for fewer epochs than we wanted.
Knowing what not to train. Our first idea was a "distress" class. We dropped it because distress isn't something a model can reliably see in a single frame.

What we learned

Most of the work in applied ML is the data, not the model. A lot of our time went into how the training data was arranged and split, and into choosing the number of epochs (full passes over the dataset) so the model wouldn’t overfit or underfit when we tested it. We also learned that the rules around the model matter as much as its accuracy. Deciding when to alert and when to stay quiet changes whether the tool is actually useful to someone relying on it in the field.

What's next for Iris

Better data. Our fine-tuned YOLO26 works well, but most of its training data was shot from drones, not eye level. Collecting and carefully labeling footage from the responder's point of view should cut down on false flags and catch more people. The fine-tuned model currently detects up to 1.8x more people than the non-tuned model. We hope to improve on this number as well as integrate wider ranges of data.

On-device detection. Right now every frame is sent to a cloud server for analysis. Disaster sites often have weak or no connectivity, so the next step is running a smaller, optimized model directly on the iPhone. That removes the network round trip, speeds up alerts, and keeps Iris working when the cell network is down.

More sensing. Adding a thermal sensor and logging each detection with location so teams can share what they've cleared.