SAR AI Agent: Drone Intelligence & Routing

Transforms drone footage into prioritized rescue routes by combining Google Cloud Gemini Vision with psychological behavior modeling, fully traced in Arize Phoenix.

Comment

Inspiration

Every year, thousands of people go missing in the wilderness, and the critical window for survival is the first 24 hours. Traditional drone search operations require human operators to stare at hours of video footage, searching for tiny clues. It is mentally exhausting and prone to human error. We were inspired to build a system that acts as a force-multiplier for Search and Rescue teams—an AI agent that never gets tired and understands exactly where a lost person is likely to travel based on their psychological profile.

What it does

The SAR AI Agent ingests live drone feeds and uses Google Cloud Gemini 2.0 Flash to detect physical clues (footprints, broken branches, campfire ash). It then feeds these detections into a Psychological Behavioral Engine based on Robert Koester's real-world "Lost Person Behavior" data. The engine calculates a dynamic Probability of Presence heatmap based on terrain attractors and generates an optimal, prioritized rescue route for ground teams.

How we built it

We built the backend using Python and FastAPI for real-time WebSocket communication. The frontend is a custom glassmorphism dashboard built with Vanilla JS and Leaflet.js for interactive mapping. For data persistence, we used MongoDB (mission data) and Elasticsearch (historical case querying). To ensure our AI routing engine was fully observable and safe for emergency response, we integrated Arize Phoenix via OpenTelemetry to trace every single decision. The entire stack is containerized using Docker Compose.

Challenges we ran into

One major challenge was the "Black Box" problem of AI. When our initial pathfinding algorithm output a rescue route, commanders had no idea why it chose those specific waypoints. Integrating Arize Phoenix allowed us to expose the exact mathematical weights and psychological profile scores (e.g., why a dementia patient was routed differently than an experienced hiker), making the AI fully transparent and trustworthy.

Accomplishments that we're proud of

We are incredibly proud of the clean, low-cognitive-load design of the dashboard. Emergency responders operate in high-stress environments, so we focused heavily on UX. We are also proud of successfully combining a multimodal LLM (Gemini) with a deterministic pathfinding algorithm (A*) to create a hybrid intelligence system.

What we learned

We learned that combining raw visual data with psychological profiling yields exponentially more accurate search areas than either method alone. We also learned how critical observability is for AI systems deployed in high-stakes, life-or-death environments.

What's next for SAR AI Agent

In the future, we plan to integrate live thermal imaging feeds, predictive weather overlays, and automatic deployment to edge devices (running the models on hardware mounted directly on the drones) to eliminate latency in low-bandwidth wilderness areas.

Share this project:

Updates