911 Operator Assistant

Being a 911 operator is very stressful, with studies showing high rates of burnout (up to 75%). That’s why an AI-powered Operator Assistant is not a “nice to have” — it’s a strategic imperative.

Comment

Inspiration

Emergency call centers are often overwhelmed, and operators must process critical information quickly under stress. We wanted to build an AI assistant that can automatically transcribe, analyze, and summarize emergency calls in real time — helping operators focus on saving lives rather than typing notes.

What it does

  • Provides a dedicated Twilio phone number that acts as a simulated 911 line.
  • When someone calls, the system:
  • Answers automatically with a scripted operator prompt.
  • Records and streams the call to the backend.
  • Transcribes the caller’s speech in real time using speech-to-text (STT).
  • Displays the live transcript and key details on a FastAPI + frontend dashboard.
  • Operators can see every caller’s transcript, time, and extracted entities like location, type of emergency, or number of victims.
  • Operators can also dispatch services like police, fire fighter, and ambulance

How we built it

  • Twilio Programmable Voice API to handle inbound phone calls and recordings.
  • FastAPI backend for call webhooks, transcription processing, and API endpoints.
  • Speech-to-text using OpenAI's Whisper for accurate call transcription.
  • WebSocket or REST API to push transcripts to the frontend dashboard in real time.
  • Frontend built with React / Next.js to visualize incoming calls, transcripts, and response times.
  • Hosted locally with ngrok for development, with potential deployment to Render / AWS.

Challenges we ran into

*Handling simultaneous callers — we had to manage transcripts per CallSid to avoid overwriting. *Webhook reliability — Twilio ends calls immediately if the server doesn’t return valid TwiML fast enough. *Integrating real-time transcription without audio lag or timeouts. *Working with audio encoding and binary data between Twilio, Python, and the STT API. *Maintaining data privacy and complying with recording consent laws.

Accomplishments that we're proud of

  • Successfully built a working phone-to-text pipeline where multiple people can call the same number simultaneously.
  • Live dashboard updates showing caller info, transcript, and detected emergency keywords.
  • Modular architecture — easy to plug in different STT or NLP models.
  • Simulated 911 operator workflow that’s realistic, fast, and privacy-conscious.

What we learned

  • How to work with Twilio webhooks and build TwiML responses dynamically.
  • The importance of async programming and background tasks in FastAPI.
  • How to process real-time audio streams and manage non-blocking transcription pipelines.
  • Collaboration between telephony, AI, and web technologies to create a unified emergency-response tool.

What's next for 911 Operator Assistant

  • Add real-time translation for multilingual callers.
  • Use NLP summarization to generate quick emergency summaries for dispatchers.
  • Integrate with GIS APIs for automatic address lookup.
  • Deploy a cloud-hosted scalable version to handle hundreds of concurrent calls.
  • Extend to SMS and video (for visual emergency reporting).

Built With

Share this project:

Updates