ResQra

A multi-agent AI system that triages, allocates, and coordinates rescue operations in real time

Comment

ResQra — Autonomous Agentic Flood Disaster Response Platform

When seconds count, traditional dispatch collapses. ResQra bridges desperate citizens and tactical rescue squads with autonomous multi-agent intelligence, 3D spatial digital twins, and strict mathematical safety boundaries.

Inspiration

Every monsoon season across South Asia and flood-prone corridors worldwide, the same tragedy unfolds: rivers like the Bagmati in Nepal’s Terai region (Rautahat District) breach their embankments, submerging entire municipalities under 6 to 10 feet of water within hours.

During these acute crises, emergency infrastructure fails catastrophically:

  1. Call lines (911/112) jam under thousands of simultaneous calls, leaving desperate families on rooftops with no response.
  2. Command centers drown in chaos, attempting to triage fragmented WhatsApp messages, unverified rumors, and scribbled spreadsheets.
  3. Pure LLM chatbots are life-threatening when deployed naively — they hallucinate non-existent rescue boats, invent false coordinates, or prioritize cases arbitrarily without respecting physical physics and capacity limits.

We were inspired to build ResQra to answer one critical question: How can we harness the language empathy of generative AI while enforcing 100% deterministic, zero-hallucination safety boundaries for life-or-death emergency dispatch?

What It Does

ResQra is an Autonomous Agentic Flood Disaster Response Platform engineered around a dual-persona paradigm:

1. The Resident Life-Line (Mobile PWA)

  • Multilingual SOS Intake: Stranded citizens submit distress calls in their native tongue (Nepali, Maithili, Bhojpuri, Hindi, or English) via text or voice.
  • Empathetic AI Guidance: Provides calming, offline-resilient survival guidance (high-ground identification, electrical safety, water purification) while extraction is underway.
  • Live Incident Tracking: Gives victims real-time visibility into their incident status (VERIFIED $\rightarrow$ PRIORITIZED $\rightarrow$ ASSIGNED $\rightarrow$ ON_MISSION).

2. The Tactical Command Center (/admin)

  • Real-Time Dispatch Radar: Instant WebSocket ingestion displaying active distress signals, calculated priority scores, and team deployments.
  • "God Eyes 3D" Digital Twin: A native WebGL 3D spatial intelligence twin built with Three.js modeling the Rautahat district basin, river confluences, 3D terrain elevation, water gauges, and live UAV/CCTV feeds.
  • Jarvis-Style Voice Agent Orb: A floating, duplex voice interface powered by Google Gemini Live and Groq, allowing coordinators to execute spoken sector queries and tactical commands hands-free.
  • Autonomous WatchAgent & 48-Hour Simulator: A background monitor that clusters distress hotspots, detects embankment breaches, triggers delay escalations, and replans routes when roads become impassable.

Safety-by-Design: The Mathematical Engine

In ResQra, LLMs read and explain, but deterministic code calculates and humans authorize. No generative model is ever permitted to calculate triage priority or assign rescue assets.

1. Deterministic Incident Priority Scoring ($P$)

The triage priority score $P \in [1, 10]$ is computed mathematically by our Python priority_engine:

$$P = \min \left( 10, \; w_u \cdot U + \sum_{k} w_v \cdot V_k + w_n \cdot \min(N, N_{\max}) + w_t \cdot \left\lfloor \frac{\Delta t}{\tau} \right\rfloor + w_r \cdot R_{\text{basin}} \right)$$

Where:

  • $U \in {1, 2, 4}$ represents base extracted urgency (LOW, MEDIUM, HIGH).
  • $V_k$ is the vulnerability weight for each flagged condition (pregnant mothers $+3$, infants/children $+2$, elderly or mobility-impaired $+2$, medical trauma $+3$).
  • $N$ is the verified victim count with weighting factor $w_n$.
  • $\Delta t$ is elapsed waiting time without dispatch, escalating priority automatically every $\tau = 15\text{ mins}$.
  • $R_{\text{basin}} \in [0, 1]$ represents live hydrological surge pressure from river gauge telemetry.

2. Constrained Fleet Allocation Metric ($S_{ij}$)

When recommending rescue squad $j$ to distress beacon $i$, our allocation_engine evaluates:

$$S_{ij} = \omega_{\text{cap}} \cdot \mathbb{I}(C_j \ge N_i) - \omega_{\text{dist}} \cdot d(x_i, x_j) - \omega_{\text{haz}} \cdot \Omega_{\text{inundation}} - \infty \cdot \mathbb{I}((i, j) \in \mathcal{M}_{\text{rejected}})$$

Where:

  • $\mathbb{I}(C_j \ge N_i)$ ensures physical boat capacity constraints are never violated.
  • $d(x_i, x_j)$ is the topological distance computed over passable road and water corridors.
  • $\Omega_{\text{inundation}}$ penalizes impassable terrain or breached dikes.
  • $\mathcal{M}_{\text{rejected}}$ is our Rejection Memory Matrix: if a human commander rejects a team recommendation, the system is mathematically forbidden from re-proposing that pairing.

How We Built It

We designed ResQra with a modern, micro-service-ready architecture separating perception, reasoning, and product state:

flowchart TB
    subgraph CITIZEN["Resident Life-Line"]
        PWA["React 19 PWA<br/>Multilingual Audio / Text"]
    end

    subgraph BACKEND["FastAPI Asynchronous Gateway"]
        AUTH["JWT + OTP Auth"]
        WS["WebSocket Event Hub"]
        DYN["AWS DynamoDB<br/>Single-Table Storage"]
    end

    subgraph AGENTS["Multi-Agent Reasoning Core"]
        INTAKE["ReportIntakeAgent<br/>Groq (Llama-3/GPT-OSS)"]
        PRIORITY["PriorityEngine<br/>Deterministic Scoring"]
        ALLOC["AllocationEngine<br/>Constrained Fleet Matcher"]
        GATE["PendingActionsStore<br/>Human Approval Gate"]
        WATCH["WatchAgent<br/>Dynamic Hotspot Sweeps"]
    end

    subgraph COMMAND["Tactical Command Console"]
        RADAR["Leaflet Tactical Map"]
        THREED["God Eyes 3D<br/>Three.js Digital Twin"]
        VOICE["Agent Orb<br/>Gemini Live Duplex Audio"]
    end

    PWA --> AUTH
    AUTH --> BACKEND
    BACKEND --> INTAKE
    INTAKE --> PRIORITY
    PRIORITY --> ALLOC
    ALLOC --> GATE
    GATE --> COMMAND
    COMMAND --> WS
    BACKEND --> DYN
    WATCH --> GATE
  • Backend: FastAPI (Python 3.12) with asynchronous routers, Pydantic v2 validation, and high-concurrency WebSockets for real-time situational radar.
  • Database & Cloud: AWS DynamoDB via boto3 utilizing an optimized single-table design for sub-millisecond lookups of teams, incidents, and audit trails. Containerized with Docker.
  • Perception & Language Layer: Groq Cloud running high-throughput open weights for near-instant dialect translation and entity extraction; Google Gemini 2.5 Flash for tactical image understanding and document analysis.
  • Real-Time Voice AI: Google Gemini Live API integrated into a custom duplex audio WebSocket proxy to power our conversational Jarvis-style command orb with native barge-in support.
  • Frontend: React 19, Vite, and Tailwind CSS with Radix UI primitives and Lucide icons.
  • 3D GIS & Spatial Twin: Three.js native WebGL renderer featuring dynamic water shaders, terrain elevation modeling, telemetry beacons, and camera orbital paths; Leaflet with clustered heatmaps and OSM Nominatim geocoding.
  • Judge Simulation Client: A standalone 48-hour event simulation harness running the ResQra-Bench automated adversarial test suite.

Challenges We Ran Into

  1. The Life-or-Death Hallucination Problem:

    • Challenge: In early iterations, LLMs would occasionally invent imaginary boats, assume infinite vessel capacities, or hallucinate GPS coordinates outside the district boundaries.
    • Solution: We instituted an absolute architectural boundary. The LLM acts purely as a structured JSON translator. All assignment logic was moved to a deterministic Python engine, and every action requires a sovereign coordinator approval card.

  2. Multilingual Terai Dialects:

    • Challenge: Flood victims frequently send frantic voice messages or text in regional dialects like Maithili and Bhojpuri mixed with colloquial Nepali (e.g., "तटबन्ध टूट गेलै, तुरंत बोट पठाउ").
    • Solution: We fine-tuned system prompts with few-shot regional emergency lexicons, enabling zero-shot extraction of victim counts, water height landmarks, and vulnerability factors.

  3. High-Performance 3D WebGL in Browser:

    • Challenge: Rendering high-polygon terrain maps of the entire flood basin alongside real-time coordinate beacons caused frame drops on standard laptop hardware.
    • Solution: We optimized the Three.js scene using custom low-draw-call geometry chunking, LOD (Level of Detail) mesh scaling, and hardware-accelerated water surface shaders.

  4. Rejection Cycling & Commander Fatigue:

    • Challenge: When coordinators rejected an automated recommendation, naive allocation algorithms would immediately re-suggest the same team.
    • Solution: We engineered an idempotent PendingActionsStore backed by a persistent RejectionMemory hash set that immediately filters rejected squads from subsequent ranking candidate pools.

Accomplishments That We're Proud Of

  • Zero-Hallucination Guarantee: Built a multi-agent system where life-critical fleet math is 100% auditable, deterministic, and unit-tested across 10 adversarial edge cases in scenario_eval.py.
  • Authentic Digital Twin: Successfully constructed the digital twin of Rautahat District, Nepal, incorporating actual hydrological data from the Bagmati and Lalbakaiya rivers, local hospital locations, and real-world boat squadrons (APF Battalion No. 11, Nepal Army Contingent).
  • Duplex Gemini Live Voice Orb: Engineered a true real-time spoken copilot that allows a disaster coordinator to speak, interrupt, and receive low-latency vocal tactical updates while keeping their hands on the keyboard.
  • 48-Hour Autonomous Simulation Engine: Built a full interactive judge client (demo-client) capable of replaying a 48-hour flood wave with dynamic obstacle injection, road cutoffs, and automated metric benchmarking.

What We Learned

  • Human-in-the-Loop is Non-Negotiable: AI should act as an unmatched cognitive amplifier that prepares high-confidence decisions, but the sovereign choice to deploy emergency resources must always remain with human incident commanders.
  • Deterministic Boundaries are the Future of Agentic AI: The most reliable agents are not monolithic LLM chains, but hybrid architectures where neural networks handle unstructured perception and deterministic code handles physics, math, and money/life constraints.
  • Local Dialect Support Saves Lives: When disaster strikes, victims don't format clean bug reports in English. Inclusive, regional language understanding is the difference between finding a family in time and missing them completely.

What's Next for ResQra

  • Direct Drone Telemetry (UAV Video Feeds): Integrating live computer vision feeds from autonomous reconnaissance drones to detect roof markers and water level rise via edge cameras.
  • Synthetic Aperture Radar (SAR) Satellite Feeds: Ingesting Sentinel-1 radar imagery to automatically update flooded road network graphs through cloud cover and heavy rainfall.
  • Field Deployments with Disaster Management Authorities: Partnering with regional civil defense squads and NGOs to pilot ResQra in annual monsoon drills across the Terai flood plains.

Built With

Share this project:

Updates