Multi-Agent AI Emergency Dispatch System

Anya scales emergency dispatch using a 3-agent AI system. It instantly answers 112 calls in 8+ Indian languages, maps threats, and dispatches units without human delays.

Comment

Here is the complete project story formatted exactly with your requested headings. It is ready to be copied and pasted directly into your submission portal!

Inspiration

When a large-scale disaster strikes, emergency response networks like India's 112 are instantly pushed past their breaking point. The core bottleneck isn't just a lack of physical resources; it is the human limitation of call operators trying to process hundreds of panicked, free-form conversations simultaneously. Precious minutes are lost just trying to extract the location and nature of the threat while dealing with panic and language barriers. We wanted to build a system that acts as a highly scalable, empathetic first responder capable of handling infinite concurrent calls without making victims wait.

What it does

Anya is a multi-agent AI emergency dispatch system. Instead of placing callers on hold during high-volume events, Anya instantly answers the phone and dynamically detects and converses in over 8 Indian languages. While the primary AI agent calms the caller and gathers details, a background agent silently listens to extract actionable JSON data (incident type, threat level, location). It then calculates the nearest emergency facility ETA and updates a live dispatch dashboard for human supervisors—all in real-time, without ever pausing the conversation.

How we built it

We moved away from a traditional linear API pipeline and utilized a concurrent multi-agent orchestration architecture:

  • The Frontline Caller Agent: Utilizing Google Gemini 2.5's native multimodal capabilities and WebRTC voice integration, this agent speaks directly to the caller. It utilizes advanced Voice Activity Detection (VAD) and audio pause detection to maintain a natural, human-like conversational cadence.
  • The Async Intelligence Agent: This agent operates asynchronously in the background. It listens to the call context to extract metadata, map the incident, and ping OpenStreetMap (OSRM) routing services for ETAs.
  • The Department Liaison Agent: Synthesizes the extracted data and formats it for specific response units (Fire, Medical, Police), pushing it directly to a React-based frontend dashboard.
  • Infrastructure: The backend is powered by FastAPI and containerized via Docker, allowing us to spin up multiple agent instances simultaneously.

Challenges we ran into

Our biggest hurdle was conversational latency. In a traditional AI voice pipeline, the time delay compounds linearly: $$T_{total} = t_{STT} + t_{LLM_inference} + t_{TTS} + t_{network}$$ During a 112 emergency, a 3-to-4 second processing delay breaks trust and heightens caller panic. We solved this by bypassing the STT/TTS pipeline entirely, feeding audio natively into the Gemini 2.5 multimodal model to enable true real-time interruption and ultra-low latency.

State synchronization was another major challenge. We had to ensure the Async Intelligence Agent could extract data and ping routing services without blocking or pausing the Frontline Agent's voice stream, which required careful decoupling of the "thinking" and "speaking" processes.

Accomplishments that we're proud of

  • Successfully eliminating the STT-to-TTS latency loop to create a genuinely interruptible, real-time voice agent.
  • Seamlessly integrating dynamic language detection for 8+ Indian languages, making the tool instantly viable for India's diverse population.
  • Building a completely decoupled, multi-agent architecture where the background data extraction does not interrupt the primary caller interaction.

What we learned

Building Anya was a masterclass in concurrent multi-agent systems. We learned how to deeply integrate real-time WebRTC voice streaming with generative models, manage complex state across isolated Docker containers, and gracefully fall back to normalized data structures when AI outputs are imperfect. We also realized how powerful low-latency AI can be when applied to critical, real-world infrastructure.

What's next for Anya-Multi-Agent AI Emergency Dispatch System

We plan to expand our language model support to cover even more regional dialects to ensure zero language barriers across the entire country. Additionally, we want to integrate the Department Liaison Agent directly with automated IoT alert systems in local firehouses and hospitals, completely automating the final step of the dispatch workflow and getting boots on the ground even faster.

Built With

Share this project:

Updates