Kontrol

Incidents
Recorded calls
Response Plans
Command center
Fleet & Dispatch

Inspiration

An emergency call arrives as a conversation, but preparing a response requires organized information: what happened, where it happened, who may be injured, and which resources could help.

Kontrol grew from a question: How could AI help a dispatcher carry those details from a call into a response plan while keeping the dispatcher in control?

The project began with recorded calls and expanded to include live phone integration. The goal stayed the same: give the dispatcher a clear connection between the caller's words, the system's recommendations, and the decision they approve.

What it does

Kontrol is a dispatcher co-pilot prototype that brings uploaded recordings and configured live inbound phone calls into one workspace.

For recordings, a dispatcher uploads audio or loads a bundled sample. For live test calls, the optional Twilio integration provides browser answering, two-way audio, mute, and hang-up controls. A separate OpenAI streaming transcription path displays caller and dispatcher speech as partial and finalized transcript segments.

After a live call ends and its transcript is finalized, it enters the same review workflow as a recording. If transcription is incomplete, the dispatcher can explicitly resolve it with reviewed text and a note. Answering or ending a call never starts the agents automatically.

Kontrol detects the incident location from the transcript and reveals it on a map. The dispatcher can check the original words, correct the address, and select the call type before starting four specialized agents:

Intake extracts incident details and identifies missing information.
Hospital researches potential destinations and their documented capabilities.
Dispatch recommends units from the supplied fleet.
Planning combines those findings into a proposed response.

The review screen brings together proposed assignments, hospital comparisons, uncertainties, and supporting details. The dispatcher can approve, reject, or request a revision. Approval rechecks recorded resource availability and constraints, saves the assignments, and displays stationary incident, vehicle, and destination markers on the map.

The current project is an exercise prototype. It does not send real responder messages, reserve hospital beds, or establish a hospital's acceptance of a patient.

How it was built

The frontend uses Next.js, React, and TypeScript, with TanStack Query for server data and MapLibre for interactive maps. Calls, incidents, response plans, fleet assignments, and agent activity share one dispatcher workspace.

The backend uses Python, FastAPI, SQLAlchemy, PostgreSQL, and PostGIS. Database-backed jobs coordinate transcription and agent execution, while PostgreSQL holds the authoritative call, incident, resource, and decision history. Uploaded audio and retained provider recordings use private storage and authenticated playback.

CREAO connects the four-agent workflow. Hospital and Dispatch run in parallel after Intake; Planning waits for both. Strict schemas and additional reference checks validate outputs before they become application state.

Live calling uses the Twilio Voice SDK and server-side OpenAI Realtime transcription over WebSockets. A separate public ingress handles signed provider callbacks and media, while browser credentials are short-lived and provider secrets stay on the server.

Docker Compose runs the application services together. Deterministic mock adapters, bundled recordings, a small demo fleet, and an explicitly loaded prepared plan provide fallback demonstration paths without requiring live provider requests.

Example CREAO agent runs

CREAO agent IDs

CREAO_INTAKE_AGENT_ID=33a03592-9b88-42e0-a4ef-c764ef1a3e2d
CREAO_HOSPITAL_AGENT_ID=cc85c669-4d51-4f19-855b-6bb2653cc143
CREAO_DISPATCH_AGENT_ID=b7682c2a-4ac2-4e0d-89a3-7ebba45194bd
CREAO_PLANNING_AGENT_ID=d4fb919b-2775-4921-8517-2e4e78ad7887

Challenges

Keeping live audio and transcript state consistent.

Live speech arrives in pieces. Partial text can change, final events can arrive late, and the phone can disconnect before transcription finishes. Handling this required timestamped segments, duplicate-event protection, explicit finalization, and a recovery path for incomplete transcripts. Phone audio is bridged independently from transcription, so an ASR failure does not itself terminate the conversation.

Turning spoken locations into usable map points.

A phrase such as “Vaness and Union” exposed the gap between transcription and address matching. Kontrol combines street-name normalization, intersection lookup, and optional address correction while preserving the original transcript. Inferred coordinates remain distinguishable from verified location information.

Making agent results reliable and readable.

An unknown patient count once caused an agent contract failure. Long narratives also made plan review difficult to scan. These problems led to clearer contracts for unknown values, compact review checks, and expandable explanations. Remote execution latency remains a challenge, even with independent stages running in parallel.

Representing only what the application knows.

The project moved away from fictional hospital capacity and automated response progression. Hospital facts now come from validated agent outputs, missing capacity and diversion remain unknown, and approved map markers stay stationary. Approval records a decision; elapsed time alone does not create an arrival or complete a response.

Accomplishments

Kontrol now connects call intake → transcription → location review → agent recommendations → dispatcher approval → mapped assignments across both recording and optional live-call paths.

It also supports the recovery steps a working prototype needs: location correction, failed-stage retries, returning an unapproved response to call-type selection, incomplete-transcript resolution, and recording deletion with retained workflow history.

The automated test suite covers permission boundaries, schema validation, concurrent actions, signed callbacks, streaming transcript behavior, and browser workflows. Provider transports are simulated in automated tests; carrier routing and browser phone connectivity require a configured real-call check.

What was learned

Building Kontrol showed that useful AI requires careful coordination around the model: clear contracts, reliable state transitions, visible uncertainty, and straightforward human correction.

It also changed the product's design. Concise review screens, meaningful resource checks, and an honest distinction between recommendations and confirmed actions became central to the experience.

What's next

The next priorities are broader testing of spoken addresses, lower agent latency, and evaluation of the review workflow with emergency communications professionals. A separately maintained, sourced hospital directory could reduce repeated research.

Further development would include provider-backed phone testing across different browsers and failure conditions. Any operational pilot would require dedicated work on production identity, recording retention, privacy, CAD integration, and validation with the participating emergency center.