You must log in to access that page

NovaCommandCenter -AI-Powered Emergency Crisis Triage Center

NovaCommand — When Every Second Counts, Nova Acts.

Comment

Inspiration

Emergency dispatch centers are among the most high-pressure, time-critical operations in public safety. Every day, dispatchers manually transcribe calls, classify incidents, fill forms, and coordinate responders—all simultaneously under extreme stress.

We asked ourselves: What if AI could handle the entire dispatch pipeline automatically, so operators could focus entirely on the human on the other end of the line?

That question became NovaCommandCenter

What It Does

NovaCommandCenter is a real-time AI-powered emergency crisis triage system that automates the entire response pipeline end-to-end:

-Intake: A dispatcher speaks or types an emergency report into the system. -Intelligence: Amazon Nova 2 Lite analyzes the transcript, corrects speech-to-text misrecognitions, classifies the emergency (type, severity, priority), and rewrites it as a professional dispatch description. -Autonomous Action: Amazon Nova Act operates the browser—filling ticket forms field-by-field, submitting to the Kanban board, navigating department portals, and marking responders "En Route"—all without a single human click. -Audit Trail: Every session is recorded as an MP4 video with AI-generated voiceover, creating a complete automated record for every incident. -Real-time Voice: Amazon Nova 2 Sonic handles high-fidelity voice recording and audio processing of the incoming calls.

How We Built It

Infrastructure

Built entirely on AWS using AWS CDK (Python). A single cdk deploy provisions the entire stack, including Lambda, ECS, API Gateway, DynamoDB, S3, and Amplify.

The "Nova" Engine Trio

  1. Nova 2 Lite (Intelligence): Deployed via AWS Lambda. It transforms raw transcripts into structured JSON (type, severity, priority, location) and generates the voiceover script for audit logs. 2.Nova Act (Automation): Runs inside an ECS Fargate container as a Flask worker. It executes a 14-step fault-tolerant workflow using Playwright. 3.Nova 2 Sonic (Voice): An ECS Fargate container exposing a WebSocket endpoint for real-time, 16kHz voice streaming.

Amazon Nova 2 Lite — Emergency Intelligence Engine

-Nova 2 Lite is the brain of NovaCommandCenter. Every time a dispatcher submits an emergency transcript, it is sent to Nova 2 Lite via Amazon Bedrock using the cross-region inference profile us.amazon.nova-2-lite-v1:0. -We crafted a structured prompt that instructs Nova 2 Lite to act as an emergency dispatch AI and return a strict JSON response containing: enhanced_description — professionally rewritten dispatch description correcting STT misrecognitions and adding context emergency_type — classified as fire, flood, medical, accident, hazmat, or other severity — low, medium, high, or critical priority — low, normal, high, or urgent location — extracted location details from the transcript personal_details — caller name, phone, address, age, medical conditions extracted from speech -Nova 2 Lite also generates the AI voiceover narration script for every Nova Act recording — a 240-260 word professional narration describing all 14 automation steps, which is then synthesized by Amazon Polly Neural into an MP3 audio file.

Amazon Nova Act — Autonomous UI Automation Engine

-Nova Act is the most critical Nova model in NovaCommand. It runs inside an Amazon ECS Fargate container (2 vCPU, 4GB RAM) as a Flask HTTP worker, triggered by the Lambda ui_automator after crisis analysis completes. -Nova Act uses Playwright under the hood with record_video=True to record the entire browser session. -We implemented a 14-step fully fault-tolerant automation workflow : Step 1 — Loads the NovaCommand dashboard with ticket parameters pre-filled in the URL Step 2 — Fills the ticket form field by field using exact element IDs ( ticket-title, ticket-severity, ticket-location, ticket-description, ticket-assignee) Step 3 — Clicks the submit button ( submit-ticket-btn) to create the ticket Step 4 — Verifies the ticket appears in the Kanban New column ( kanban-new) Step 5 — Navigates to the department portal URL with the correct department parameter Step 6 — Waits 20 seconds for the portal to poll DynamoDB and display the ticket Step 7 — Finds the ticket card and clicks the green Accept & Respond button Step 8 — Fills responder name in the modal ( responder-name input) Step 9 — Sets ETA to 5 minutes ( eta-minutes input) Step 10 — Clicks the Accept & Respond button ( accept-respond-btn) Step 11 — Clicks the En Route button ( en-route-btn) and waits for Status Updated confirmation Step 12 — Navigates back to the dashboard and verifies ticket moved out of New column Step 13 — Opens the ticket modal, fills detailed response notes with responder name, ETA, dispatch time, and incident summary, then saves Step 14 — Calls the tickets API to persist the Assigned status to DynamoDB, waits for Kanban to re-poll and visually confirm the Assigned column -Each step is wrapped in a _safe_act helper that catches exceptions and continues recording — a failed step never cuts the video. After all context managers exit, we poll the webm file size every second until it stabilises (no growth for 3 consecutive seconds) before converting to MP4, ensuring Playwright has fully flushed all frames to disk. -The final output is a silent MP4 (full browser recording) and a separate MP3 voiceover — both uploaded to S3 with presigned URLs and stored on the DynamoDB ticket record for playback in the ticket modal.

Amazon Nova 2 Sonic — Voice Processing

-Nova 2 Sonic runs as a separate ECS Fargate container exposing a WebSocket endpoint. -When the dispatcher enables Nova Sonic Mode in the frontend, audio is streamed in real-time from the browser microphone at 16kHz as Int16 PCM chunks via WebSocket. -Nova Sonic processes the voice input and captures the emergency call audio for transcript generation, enabling a hands-free dispatch experience.

Challenges We Ran Into

Nova Act Element Targeting Across Two Apps

Nova Act had to autonomously operate two completely separate web applications — the NovaCommandCenter dispatcher dashboard and the Emergency Department Portal — in a single session. -Every step instruction had to reference exact HTML element IDs. Any mismatch between the Nova Act prompt and the actual DOM caused silent failures. -We had to carefully audit every element ID across IntakePanel.jsx, KanbanBoard.jsx, TicketModal.jsx, and ResponseModal.jsx and write Nova Act instructions precise enough that the model could find elements even when they were inside scrollable modals or off-screen.

Nova Act Timing — Coordinating with Live APIs

-Nova Act automation runs against a live AWS backend . The department portal polls DynamoDB every 3 seconds. After ticket submission, Nova Act had to wait long enough for: -The ticket to be written to DynamoDB by the form submission -The portal to poll and render the ticket card -The handleAccept API call to complete and the modal to transition to step 2 -The Kanban board to re-poll after move_ticket_to_assigned -Too short a wait meant Nova Act was clicking on elements that hadn't appeared yet. Too long wasted recording time. We tuned each sleep through multiple test runs.

Nova 2 Lite Cross-Region Inference Profile

-Nova 2 Lite is only accessible via the cross-region inference profile ( us.amazon.nova-2-lite-v1:0) — not the direct model ARN. -Calling the standard amazon.nova-2-lite-v1:0 model ID directly from us-east-1 resulted in model access errors. We had to use the us. prefixed inference profile which routes requests across US regions automatically for higher availability.

Nova Act Workflow Session Management

-Nova Act uses a Workflow context manager that must wrap the NovaAct context manager. If the workflow session expired or the Nova Act API rate-limited mid-automation, the entire 14-step sequence would fail silently. -We implemented a _safe_act wrapper that catches all exceptions per step and logs them without raising — ensuring the recording always continues to completion regardless of individual step failures:

def _safe_act(nova, instruction, step_name, steps_completed, max_steps=20): try: nova.act(instruction, max_steps=max_steps) steps_completed.append(f"{step_name}: success") except Exception as e: steps_completed.append(f"{step_name}: failed — {str(e)[:120]}") # recording continues regardless _time.sleep(3)

Accomplishments That We're Proud Of

-Zero-Touch Dispatch: Built a pipeline that goes from voice input to responder assignment with zero manual data entry. -Complex Agency: Nova Act successfully executes 14 autonomous browser steps across two separate web applications. -Automated Evidence: Created a first-of-its-kind video audit trail with AI narration for legal and training accountability. -Infrastructure as Code: 18 AWS services provisioned seamlessly with a single command.

What We Learned

-Agentic Reasoning: Nova Act is genuinely capable of reasoning about multi-step UI workflows—finding elements and recovering from UI failures gracefully. -Intent over Syntax: Large Language Models (like Nova 2 Lite) are better at understanding "Emergency Intent" than standard spellcheckers. -Stability is Key: In headless environments, you can't rely on visual cues; you must implement robust polling and state-checks to ensure the automation stays on track.

What's Next for NovaCommand

-Multi-language Support: Analyzing calls in regional languages for wider accessibility. -IoT Integration: Automatically creating tickets triggered by flood sensors or fire alarms. -Live Map View: Real-time incident plotting with live responder GPS tracking. -Nova Multimodal: Enabling image analysis so photos from the scene automatically update the dispatch description.

Built With

Share this project:

Updates