AEGIS — AI Emergency Guidance & Incident System

You can't manage what you can't see.

Inspiration

During a mass-casualty emergency, incident commanders face an extraordinary challenge: coordinating responders while maintaining an accurate picture of a rapidly evolving scene.

Information arrives through radio calls, field reports, and individual assessments. Casualties may move, conditions can deteriorate, and critical information can become fragmented, delayed, or outdated.

This isn't just a hypothetical problem. Real emergency-response investigations have documented these challenges repeatedly.

  • 2017 Las Vegas shooting: The 1 October After-Action Report documented significant challenges involving casualty tracking, coordination, and patient accountability. Observation 20 specifically identified difficulties tracking patients and the hospitals to which they were transported.
  • 2022 Uvalde school shooting: The U.S. Department of Justice's 2024 critical incident review identified serious failures in incident command, leadership, coordination, and emergency communications, including problems affecting the medical response.
  • 2023 Lewiston mass shooting: Maine's 2024 independent commission investigated the tragedy and the circumstances surrounding the emergency response, highlighting the importance of effective interagency preparedness and coordination.
  • 2023 Maui wildfires: Maui Emergency Management Agency's 2025 after-action report examined how infrastructure damage and information-sharing difficulties affected situational awareness, emergency communication, and evacuation coordination.

Although these incidents differ considerably, their investigations reinforce an important lesson: effective emergency response depends on reliable information reaching the right people at the right time.

During HackPrinceton 2026, our team also conducted a live interview with a Princeton firefighter to better understand emergency-response operations.

We discussed incident-command coordination, casualty accountability, incomplete situational awareness, changing conditions, and the importance of minimizing additional responder workload.

A particularly important insight from our conversation was the need to distinguish what has actually been observed, what has changed, and what remains unknown.

A person disappearing behind an obstruction does not necessarily mean their condition changed. An outdated observation should not be presented as current information. And technology should help organize decisions without taking operational authority away from trained personnel.

The interview was reconstructed from memory rather than recorded verbatim, and the resulting insights informed our engineering decisions.

Together, this field research and the documented challenges in emergency-response literature inspired us to build AEGIS — AI Emergency Guidance & Incident System.

We wanted to investigate one question:

What if incident commanders could see a continuously updated picture of an emergency, receive intelligent responder recommendations, and coordinate connected devices without surrendering human control?

What it does

AEGIS is a human-supervised, AI-powered emergency-response coordination platform combining computer vision, mathematical optimization, trained machine learning models, real-time situational awareness, and connected responder hardware.

Built as a simulation-first prototype, AEGIS demonstrates how visual observations, responder assignments, casualty records, and physical devices can operate through one coordinated system.

1. Computer Vision & Persistent Casualty Tracking

AEGIS uses live camera feeds to detect and track people within a simulated emergency scene.

Rather than treating every frame independently, the system maintains persistent observation histories and analyzes observable changes, including movement and posture.

It tracks detection confidence, observation freshness, and uncertainty, helping distinguish between:

  • A person whose position or behavior has visibly changed.
  • A person temporarily obscured by an obstacle.
  • A stale observation that can no longer establish current conditions.
  • A tracking failure rather than a confirmed casualty event.

This distinction matters because uncertain camera observations should never be treated as established medical facts.

Computer vision does not autonomously assign medical priorities. Human operators confirm casualties and establish simulated assessments before those records influence responder coordination.

2. Mathematical Optimization — OR-Tools CP-SAT & A*

At AEGIS's core is a constraint-based responder-assignment engine built using Google OR-Tools CP-SAT.

Once human operators establish casualty priorities, the backend computes responder-to-casualty assignments using defined operational constraints, including responder availability, priority coverage, and route feasibility.

An A* pathfinding algorithm computes navigable routes across the simulated scene, accounting for obstacles and blocked corridors.

Instead of simply choosing the closest available responder for each casualty, AEGIS evaluates assignment combinations across the incident and produces a structured proposal.

The operator can review the recommended assignments before any dispatch occurs.

3. Neural Assignment Intelligence — MLP & Graph Neural Network

To investigate how learned models compare with mathematical optimization, we independently developed two machine learning approaches to responder allocation.

Pairwise Multilayer Perceptron (MLP)

Our MLP learns to score potential responder-casualty pairings using features from the simulated assignment context.

Bipartite Graph Neural Network (GNN)

Our GNN represents responders and casualties as two groups of nodes connected by potential assignment relationships.

This allows the learned model to reason over the structure of the assignment problem rather than treating every responder-casualty pair entirely independently.

Both models were trained and evaluated using synthetic assignment scenarios.

Their predictions are displayed alongside the OR-Tools recommendation, allowing AEGIS to compare:

  • Selected responder-casualty assignments.
  • Priority coverage.
  • Total route distance.
  • Constraint feasibility.
  • Inference and processing time.

This produces a live research comparison between exact constraint-based optimization and learned neural assignment strategies.

The distinction is intentional: our MLP and GNN are advisory models, not autonomous dispatch authorities.

OR-Tools remains responsible for the authoritative optimization proposal, and human approval is always required before dispatch.

4. Real-World Machine Learning — XGBoost Trained on 377,606 NYC EMS Incidents

Beyond synthetic responder-assignment experiments, we wanted to investigate how real emergency-response data could support predictive modeling.

We obtained 377,606 historical incidents from New York City's official EMS Incident Dispatch Data, covering January through March 2023.

Using XGBoost, we trained a regression model to estimate the elapsed time between an incident's creation and the first responder assignment.

To better evaluate generalization over time, we used chronological data partitions rather than a randomized train/test split.

Dataset partition Incidents Period
Training 245,744 January 1 – February 28, 2023
Validation 62,291 March 1 – March 15, 2023
Held-out testing 67,798 March 16 – March 31, 2023

After validity filtering and partition-boundary exclusions, these partitions contain 375,833 incidents.

Held-out test performance:

Metric Temporal baseline XGBoost
Mean Absolute Error 82.63 seconds 78.39 seconds
Root Mean Squared Error 548.90 seconds 410.28 seconds

On the full chronological holdout, this represents approximately:

  • 5.1% lower mean absolute error.
  • 25.2% lower root mean squared error.

However, deeper evaluation revealed an important limitation.

Much of the aggregate improvement came from a small subgroup of unusual severity-8 incidents. Excluding these cases reduced the improvement substantially, and the model performed worse than the baseline for some common incident groups.

Our generalization audit also investigated outlier influence, feature availability, chronological leakage, and subgroup performance.

Rather than claiming broad emergency-response improvements, we documented both the model's strengths and its limitations.

AEGIS uses genuine trained XGBoost weights, not simulated prediction values.

The model is exposed as an optional, read-only historical research advisory. It does not determine casualty urgency, predict indoor movement time, or control responder assignments.

This work taught us that responsible machine learning requires understanding when a model fails, not just celebrating when its aggregate metrics improve.

5. Connected Physical Hardware — Arduino UNO R4 WiFi & UNO Q

AEGIS extends beyond the browser through two physical Arduino responder devices:

  • Arduino UNO R4 WiFi — Responder Alpha
  • Arduino UNO Q — Responder Bravo

Each device represents an individual responder and communicates with AEGIS through a hardware integration bridge.

When the operator approves and dispatches an assignment, the corresponding device receives its own instructions.

Responders can acknowledge and complete tasks using physical buttons, transmitting those events back to the backend.

During physical integration testing, both boards successfully received distinct assignments during the same simulated dispatch.

Real button presses generated separate acknowledgment and completion events, which were recorded by the backend under their respective device identities.

This demonstrates the connection between AI-assisted coordination software and physical responder hardware.

6. Dynamic Route Invalidation & Human-Supervised Replanning

Emergency environments change.

If a corridor becomes obstructed after responders have been dispatched, AEGIS checks whether the simulated obstacle invalidates an existing route.

When it does, the system:

  • Flags the route as invalidated.
  • Preserves the original dispatched route as historical information.
  • Calculates an advisory alternative route.
  • Displays an operator-facing warning.
  • Requires renewed human authorization before revised dispatch instructions.

AEGIS never silently redirects physical responders based on an unapproved recommendation.

This separation between detecting a problem, recommending an alternative, and authorizing action is central to our design.

7. Real-Time Command Center & Voice Intelligence

All these systems connect through the AEGIS Command Center.

Our dashboard consolidates:

  • Live camera feeds and computer-vision observations.
  • A spatial scene map and responder positions.
  • Casualty records, assessments, and observation histories.
  • Optimized assignment proposals and navigable routes.
  • Responder availability and hardware acknowledgment status.
  • Neural optimization comparisons.
  • Dynamic route warnings and operational alerts.
  • Historical NYC EMS machine learning research.

We also developed Ask AEGIS, an ElevenLabs-powered, read-only voice assistant designed to answer natural-language questions about the current simulation.

The assistant provides grounded incident information without requiring operators to manually search through every dashboard panel.

It cannot approve plans, modify medical assessments, dispatch responders, or control hardware.

How we built it

We developed AEGIS as independently engineered subsystems connected through a shared Python backend and synchronized simulation state.

Live camera observations enter the system through a computer-vision pipeline, where operators can confirm casualty records and establish priorities. OR-Tools CP-SAT computes constrained responder assignments, while A* generates routes through the simulated environment. Our PyTorch MLP and GNN models independently produce advisory assignment comparisons.

A Next.js Command Center displays the evolving incident through REST APIs and WebSockets. After human approval, Python hardware bridges deliver assignments to Arduino devices and relay acknowledgment and completion events back into the dashboard.

We also integrated an ElevenLabs-powered situational-awareness assistant and a separately trained XGBoost model for historical NYC EMS research.

Throughout development, we deliberately separated observations, predictions, optimization, human decisions, and physical actions.

Our end-to-end workflow is:

Observe → Confirm → Assess → Optimize → Approve → Dispatch → Acknowledge → Adapt

Challenges we ran into

Integrating Multiple Independent Systems

AEGIS combines computer vision, optimization, neural inference, voice interaction, frontend visualization, and embedded hardware.

Developing these components in parallel required maintaining compatible APIs, consistent responder and casualty identifiers, synchronized simulation revisions, and reliable state transitions.

Our biggest integration challenge was making these systems operate as one coherent platform rather than a collection of isolated demonstrations.

Reliable Physical Hardware Communication

Integrating two different Arduino platforms required distinct firmware configurations, device identities, communication bridges, and acknowledgment protocols.

We successfully demonstrated simultaneous physical assignments and acknowledgment/completion events, but also encountered intermittent UNO Q network connectivity.

These issues reinforced the importance of resilience, communication monitoring, and reliable fallback behavior.

Reasoning Under Incomplete Visual Information

A camera cannot observe everything.

Someone disappearing behind an obstruction should not automatically be classified as having moved, fallen, or experienced a medical emergency.

We therefore emphasized observation histories, confidence estimates, stale-track detection, and explicit human confirmation.

Combining Learned Models with Hard Constraints

An ML model may generate a plausible assignment that violates operational constraints.

Our solution was to keep neural inference advisory-only and preserve the constraint-based optimizer as the authoritative recommendation engine.

Human approval remains a separate step.

Evaluating Real-World Data Responsibly

Our NYC EMS XGBoost model demonstrated how aggregate metrics can hide meaningful weaknesses.

Although the model improved overall held-out error, subgroup and outlier analyses showed that improvements were not distributed evenly across incident categories.

We conducted additional generalization and leakage audits to better characterize what the model actually learned.

Balancing Functionality with Responder Workload

Our firefighter interview highlighted the importance of providing useful information without overwhelming the people responding to an incident.

This influenced our decision to emphasize concise incident summaries, observation freshness, visible uncertainty, and human-controlled operations.

Accomplishments that we're proud of

  • Real stakeholder engagement: Conducted a live interview with a Princeton firefighter to understand incident-command coordination, patient accountability, and situational-awareness challenges.
  • Research-grounded problem definition: Connected the project to documented challenges identified in emergency-response after-action reports.
  • Integrated AI coordination: Built a simulation platform connecting computer vision, optimization, neural inference, a real-time dashboard, voice interaction, and physical responder hardware.
  • Mathematical optimization: Implemented OR-Tools CP-SAT assignment optimization with A* route planning and dynamic route invalidation.
  • Neural learning: Developed MLP and bipartite GNN models for learned responder-assignment comparisons.
  • Real-world ML training: Trained and evaluated XGBoost on over 377,000 NYC EMS incidents and conducted a detailed generalization audit.
  • Physical hardware integration: Demonstrated distinct responder assignments and acknowledgment/completion events on two Arduino devices.
  • Human-supervised decision-making: Preserved human authority over casualty confirmation, medical priority, assignment approval, and dispatch.
  • Uncertainty-aware observations: Implemented persistent tracking, observation histories, stale-track handling, and confidence indicators.

What we learned

Building AEGIS taught us that the most valuable AI system isn't necessarily the one making the most decisions autonomously.

In high-stakes environments, usefulness depends on reliability, transparency, uncertainty awareness, and preserving human judgment.

We learned how mathematical optimization and neural networks can complement one another, how camera-based observations can be integrated into persistent state, and how a software recommendation can be connected to physical responder hardware.

Our NYC EMS research also showed that model evaluation cannot stop at one accuracy metric. Distribution shifts, outliers, subgroup differences, and prediction-time feature availability all matter.

Perhaps our biggest insight came from combining field research with technical development:

Knowing what remains uncertain can be just as important as knowing what has been observed.

An incident-command system should communicate that uncertainty rather than conceal it.

Our conversation with a Princeton firefighter reinforced the importance of reducing information fragmentation and responder workload without interfering with established emergency-response procedures.

What's next for AEGIS

AEGIS is an early-stage, simulation-first engineering and research prototype.

We plan to explore:

  • More robust multi-camera tracking and observation continuity across occlusion.
  • Better uncertainty calibration for computer-vision observations.
  • More realistic and diverse synthetic emergency-response scenarios.
  • Larger-scale evaluation of neural assignment models against exact optimization.
  • Improved connectivity resilience for hardware devices and distributed systems.
  • Additional environmental sensors and changing-scene detection.
  • Structured usability studies and further interviews with emergency-response professionals.
  • Research into whether uncertainty-aware dashboards improve situational awareness while reducing operator workload.

Before any real-world emergency use, AEGIS would require substantial operational validation, security hardening, professional stakeholder evaluation, and rigorous safety testing.

Our long-term vision is to help emergency personnel spend less time reconstructing fragmented information and more time making informed decisions.

AEGIS doesn't replace incident command. It helps incident command see the bigger picture.


References

  1. 1 October After-Action Report (2018). Las Vegas Metropolitan Police Department and Clark County Fire Department. See Observation 20 on patient accountability and tracking.
    https://www.theiacp.org/sites/default/files/MVAI/Library%20of%20Resources/Las%20Vegas%20Shooting%20After-Action%20Report.PDF

  2. Critical Incident Review: Active Shooter at Robb Elementary School (2024). United States Department of Justice, Office of Community Oriented Policing Services.
    https://cops.usdoj.gov/html/dispatch/02-2024/uvalde_critical_incident_review.html

  3. Final Report of the Independent Commission to Investigate the Facts of the Tragedy in Lewiston (2024). State of Maine.
    https://www.maine.gov/icl/sites/maine.gov.icl/files/2024-08/Final%20Report%20Of%20The%20Independent%20Commission%20To%20Investigate%20The%20Facts%20Of%20The%20Tragedy%20In%20Lewiston.pdf

  4. Maui Wildfires 2023 After-Action Report (2025). Maui Emergency Management Agency.
    https://www.mauicounty.gov/DocumentCenter/View/151355/MEMA-2023-Wildfire-After-Action-Report

  5. EMS Incident Dispatch Data. NYC Open Data, official historical EMS dispatch dataset.
    https://data.cityofnewyork.us/Public-Safety/EMS-Incident-Dispatch-Data/76xm-jjuj

  6. Princeton Firefighter Field Interview (November 2026). AEGIS team, HackPrinceton. Live interview; retrospectively reconstructed paraphrases, not a verbatim transcript. Not a formal Princeton Fire Department endorsement.

Built With

Share this project:

Updates

Submission history