Inspiration

When people call 911, they want it picked up on the first ring, usually with a good and "urgent" reason. But people often have to wait longer than expected due to a lack of 911 operators especially during the pandemic of infectious disease just like now.

In South Korea, where we live, one sick employee has taken out the entire call center. Of 97 confirmed cases, 94 were working on the same floor with 216 employees, translating to an attack rate of 43.5%. It's a lot bigger than the household secondary attack rate among symptomatic case-patients which is only 16.2%. So It's clear that call centers are vulnerable to infectious diseases due to the nature of the jobs that don't have a large space between people and have to keep talking. Also, it's important to isolate the suspected patients in the early stage to block further transmission in crowded work settings. But what if a 911 operator gets infected and the entire center goes into quarantine? Then who will answer the calls?

What it does

AI Assistant for 911 converts your voice directly into text and automatically extracts key information with an AI model that has learned emergency call scenarios, significantly reducing the speed of call managing.

  • Relay the voice from operator to caller, and vice versa
  • Recognize the voice and extract the valuable information by using Wit.ai STT API
  • Visualize the information on a browser by using React

How we built it

Wit.ai provides most of the functions we want to implement. Thanks to Wit.ai, The only part that we spent on NLP is adding a separate model to Wit.ai by recognizing special entities such as emergencies and injuries.

The STT function in Wit.ai takes the part of converting voice data into text, and the built-in NLP function can extract and classifies entities like place and time.

Challenges we ran into

Our biggest challenge was streaming audio between browser and server. There was several methods to achieve this goal; especially using WebRTC and establishing the call. But it was too hard to achieve quickly; we have to construct STUN/TURN server to properly use this protocol.

So we convert the audio to a series of buffers and streamed it to server by using socket.io. We tried to transmit our audio buffer as soon as possible to the opposite side, and we achieved only 1s latency while the call.

We decided to stop developing the advanced features for this since this is a demo, but we think that we can construct the appropriate call structure by using the proper infrastructure.

What's next for 911 Assistant

If this gives us a certain level of accuracy, we can introduce it into the queueing system where a simple scenario would allow us to proactively extract key information from requests waiting to be answered, thereby reducing a huge amount of time to manage emergency calls.

Reference

Share this project:

Updates