Hawkeye

Live, voice-controlled 3D command center.

Comment

Inspiration

Indonesia faces devastating floods every year. In 2025 alone, over 1,000 people died in flood-related disasters across the archipelago. First responders in Jakarta — a metro area of 11 million people — still rely on 2D paper maps, radio chatter, and gut instinct to make life-or-death decisions about evacuations, hospital routing, and resource allocation.

There is no system that:

  • Predicts what happens when water rises 2 meters
  • Knows which hospitals will be cut off before it happens
  • Disagrees with a commander when an evacuation route is unsafe
  • Speaks proactively when critical thresholds are exceeded

What it does

The Multi-Agent Architecture

Hawk Eye is built on Google's Agent Development Kit (ADK) with four specialized sub-agents:

  1. Perception — Analyzes drone footage and satellite imagery for structural damage, crowd movements, and flood extent
  2. Analyst — The intelligence engine. Queries 2.6 million historical flood events, computes cascade consequences, evaluates route safety
  3. Predictor — Uses Nano Banana 2 (Gemini 3.1 Flash Image) to generate visual risk projections of future flood scenarios
  4. Coordinator — Executes actions: sends emergency alerts via Gmail, generates evacuation routes, logs incidents

How we built it

  • Backend: Python FastAPI with Google ADK bidi-streaming pattern
  • Live API: WebSocket at /ws/{user_id}/{session_id} with three concurrent tasks (upstream, downstream, proactive monitoring)
  • Voice: Gemini 2.5 Flash Native Audio with barge-in support, proactive audio, and session resumption
  • Database: BigQuery with clustered geospatial tables + Firestore for real-time sensor data
  • Visualization: React + Vite + CesiumJS with Google 3D Tiles API
  • Satellite Analysis: Google Earth Engine with Sentinel-1 SAR for flood extent detection We engineered HawkEye using a cutting-edge, highly parallelized microservices architecture to ensure it could handle massive data streams in real-time.
  • The Brain: We utilized the Gemini Live API and Google's Agent Development Kit (ADK) to build a lightning-fast Bidi-Streaming WebSocket backend. This powers the "Hawk Eye Commander"—a root AI that orchestrates a swarm of specialized sub-agents (Perception, Analyst, Predictor, and Coordinator).
  • The Intelligence: We leveraged Gemini 2.5 Flash for blazing-fast reasoning and Nano Banana 2 (Gemini 3.1 Flash Image Preview) to procedurally generate striking, photorealistic visual projections of future flood states.
  • The Data Engine: We integrated Google Earth Engine for live Sentinel-1 SAR flood detection. We loaded massive Groundsource datasets into Google BigQuery, utilizing advanced spatial joins (ST_DWITHIN, ST_INTERSECTS) to calculate exact demographic and infrastructural impacts in milliseconds.
  • The Visuals: We built a futuristic, dark-mode React frontend integrating CesiumJS and Google Maps Photorealistic 3D Tiles to render the operational context interactively.

Challenges we ran into

Building a system this ambitious in a hackathon timeframe was like jumping out of a plane and assembling a parachute on the way down.

  • Sub-second Voice Latency: Wiring up a full-duplex audio pipeline from the browser microphone, through our WebSocket server, into the Gemini Live API, and streaming the response back without stuttering was incredibly complex.
  • The Math of Chaos: Designing the intelligence required pushing BigQuery to its limits. We had to write complex geospatial SQL queries that could calculate newly at-risk infrastructure based on dynamic water level changes ($\Delta h$) in real-time, all while the AI was "thinking."
  • Orchestrating the Swarm: Making four independent AI agents talk to each other, share context, and disagree safely (e.g., the Analyst agent rejecting an unsafe evacuation route proposed by the Coordinator) required meticulous prompt engineering and tool design.

Accomplishments that we're proud of

We built a system that feels like magic. We are immensely proud of our Cascade Engine. Most AI apps just summarize text; ours calculates physical consequences in the real world. We successfully integrated 17 different Google Cloud APIs and services into a single, cohesive masterpiece. Hearing the HawkEye AI physically speak to us, warn us about a compromised evacuation route, and autonomously project the new safe path on a 3D globe was a jaw-dropping moment for our entire team.

What we learned

We learned that the true power of Large Language Models isn't in chat boxes—it's in orchestration. When you give an AI like Gemini access to deterministic, world-class tools like BigQuery spatial functions and Earth Engine, it transcends being a chatbot and becomes a true synthetic intelligence. We also learned that voice is the ultimate user interface for high-stress, mission-critical environments where your eyes and hands are busy.

What's next for Hawkeye

This is just the beginning. Next, we plan to scale HawkEye globally, moving beyond Jakarta to monitor high-risk zones worldwide. We want to integrate live IoT sensor networks (water levels, seismic activity) directly into the agent's data stream. Ultimately, we envision HawkEye as the standard operating system for emergency response agencies across the globe, saving thousands of lives through predictive, autonomous logistics.

Built With

Share this project:

Updates