RescueLens

RescueLens turns live disaster alerts and drone evidence into safer, human-approved rescue actions, with Arize catching computer vision failure risks before dispatch.

Comment

Inspiration

During a disaster, responders are not waiting for another chatbot. They are trying to make fast decisions from messy information: live alerts, drone images, blocked roads, damaged structures, changing weather, and uncertain model predictions.

That is the moment RescueLens is built for.

The idea started from one question:

What if an AI agent could help responders act faster, but also show when its own computer vision might be wrong?

In emergency response, a confident AI answer is not enough. A road can look passable when floodwater glare is hiding damage. A small object can be missed in smoke or low light. A route can look safe until someone checks the risk behind the image.

RescueLens combines Gemini reasoning, Arize Phoenix MCP failure analysis, and human approval so the system does more than generate a recommendation. It helps responders understand whether that recommendation is safe enough to trust.

What It Does

RescueLens is a supervised disaster-response agent.

It helps response teams move through one mission flow:

live disaster signal -> drone evidence -> Gemini briefing -> Arize risk review -> human-approved action

With RescueLens, a user can:

  • pull live public disaster signals from NOAA/NWS, USGS, and NASA EONET
  • search and inspect incident locations on a map
  • attach drone or field imagery to the mission
  • analyze visual evidence for hazards, road risk, damage, and possible rescue signals
  • inspect object detection, segmentation, heatmaps, and safety-route indicators
  • generate responder briefings, mission plans, and recommended actions with Gemini
  • use Arize Phoenix MCP to surface risky computer vision failure patterns
  • create reviewable artifacts such as mission reports, dispatch tasks, route closures, and safety plans

The key idea is simple:

RescueLens does not just recommend action. It checks model risk before responders act.

How We Built It

RescueLens is a deployed web application running on Google Cloud Run with a Node.js backend and a mission-focused dashboard.

The system connects several pieces into one operational flow:

  • Gemini handles image reasoning, mission planning, responder briefings, command planning, and voice responses.
  • Google Agent Platform / Agent Builder supports the managed-agent handoff and runtime interaction.
  • Arize Phoenix MCP provides the reliability loop: failure slicing, eval comparison, and improvement recommendations.
  • Computer vision telemetry represents classification, object detection, segmentation, embeddings, drift clusters, monitors, and evaluator-style outputs.
  • Live public data APIs give the agent real incident context instead of a purely scripted scenario.
  • Human approval controls keep reports and response actions reviewable before they are used.

The most important design choice was making Arize part of the decision flow, not just a status badge. When the system detects a risky vision condition such as low-light water glare, the agent can shift from a dispatch recommendation to a safer action: request another drone angle, mark the route as uncertain, or require human review.

Challenges We Ran Into

The hardest part was not calling one model or showing one map. The hard part was connecting the whole mission loop in a way that felt real:

live incident data, drone evidence, Gemini planning, Arize failure analysis, and human approval

All of those pieces had to work together inside a short demo. If the flow was too technical, the value was hard to understand. If it was too simple, it looked like a static dashboard. We had to keep improving the UI until the story was clear.

Trust was another major challenge.

In disaster response, the problem is not only whether an AI can produce an answer. The problem is whether a human can quickly inspect the evidence, understand the uncertainty, and decide whether the next action is safe.

That is why RescueLens shows risk signals, failure patterns, and approval steps directly in the workflow.

Accomplishments That We're Proud Of

We are proud that RescueLens moves beyond chat.

It can start from a live incident, connect that incident to drone evidence, run Gemini planning, invoke the Arize reliability loop, and produce an operational artifact for human review.

The most important moment in RescueLens is when model risk changes the recommendation.

For example, if the visual evidence suggests a route may be passable, but Arize surfaces a risky failure slice such as floodwater glare, the system can avoid overconfident dispatch and move the mission toward human review or a second drone pass.

That is the difference between a demo that only detects things and a workflow that helps people act more safely.

We are also proud that the final design keeps responders in control. RescueLens is built around supervision, not automation for its own sake.

What We Learned

We learned that useful disaster AI needs three things working together:

  1. live context from the real world
  2. strong reasoning from an agent model
  3. observability that explains when the model might fail

We also learned that agents become much more useful when they produce reviewable work, not just text answers.

Reports, route recommendations, safety plans, and dispatch tasks give humans something concrete to inspect, approve, reject, or revise.

Most of all, we learned that model risk should not be hidden in a log file. In high-stakes workflows, it needs to be visible at the moment a decision is being made.

What's Next For RescueLens

Next, we want to make RescueLens closer to a real emergency operations tool.

Planned improvements include:

  • connecting to more official emergency data feeds
  • supporting live drone video streams
  • adding multi-agent roles for search, route safety, medical triage, and infrastructure recovery
  • sending approved artifacts to real incident management systems
  • expanding Arize monitoring with production datasets, richer evals, and long-term drift tracking
  • improving mobile field use for responders outside the command center

The long-term vision is a supervised AI operations layer for disaster response: fast enough to help in the moment, transparent enough to trust, and always under human control.

Built With

  • arize-phoenix-mcp
  • arize-style-computer-vision-observability
  • css
  • docker
  • gemini-api
  • google-agent-platform-/-agent-builder
  • google-artifact-registry
  • google-cloud-build
  • google-cloud-run
  • google-secret-manager
  • html
  • leaflet.js
  • model-context-protocol
  • nasa-eonet-api
  • noaa/nws-alerts-api
  • node.js
  • open-meteo-geocoding-api
  • usgs-earthquake-geojson-api
Share this project:

Updates