Inspiration

We were inspired by the recurring floods in the Philippines and the uncertainty families face when loved ones go missing. Rising water can leave people stranded on rooftops, trapped in isolated communities, or separated from their families. Floodwater, debris, and poor visibility can make survivors difficult to locate, even while rescue teams work urgently to reach them.

The possibility that someone could survive a flood but remain unfound when help is needed most motivated us to build Aegis. Searching large areas and continuously monitoring drone footage places a heavy burden on rescuers, especially when every minute matters.

We wanted to help rescue teams identify possible survivors, recognize hazards, and turn those observations into clearer response decisions. Aegis is our attempt to shorten the time between someone needing help and rescuers knowing where to look, while keeping human commanders in control.

What it does

Aegis processes drone footage, identifies potential survivors and hazards, evaluates confidence and location information, and selects a response: record a supply-dispatch intent, propose a ground-rescue reroute, or escalate the situation for human review.

Its Command Center brings together annotated video, target information, confidence scores, hazard status, available geographic coordinates, and the current spatial state. Operators can inspect execution traces and activate a commander override to block subsequent autonomous action submissions.

The current prototype records decisions and action intents. It does not physically fly drones or move rescue vehicles.

How we built it

We developed Aegis as a Python monorepo with separate components for video ingestion, perception, spatial reasoning, decision-making, and operator visibility.

OpenCV 5 handles video acquisition, preprocessing, segmentation, classical detection, and frame encoding. We also integrated an optional YOLO11n person-and-boat detector trained using SeaDronesSee and MOBDrone data. Person detections map to potential survivors, while boat detections map to hazards.

FastAPI provides the central integration bus. Strict Pydantic contracts keep data consistent between components, and WebSockets deliver matched video frames, spatial assessments, and action traces to the browser dashboard. Bounded queues and dedicated decoding work prevent slow consumers from building an ever-growing video backlog.

We implemented MCP tools for reading vision states, assessing blur, reanalyzing frames, and recording rescue actions. The dashboard uses HTML, CSS, and JavaScript, with an optional Streamlit launcher.

We packaged the classical detector with Docker and ran it on ECS Fargate in ap-southeast-2, reading Skydio’s public demo feed. The task definition in the repository still uses the mock source and is not marked deployment-verified. The trained detector was scored on local hardware and is not in that cloud image.

Challenges we ran into

Small and distant swimmers were difficult to detect. Reflections, wake foam, boats, and background structures also created false positives. Our evaluation showed stronger boat detection than person detection, making swimmer recall an important remaining challenge.

Keeping asynchronous components synchronized was another major task. We needed to ensure that bounding boxes, confidence assessments, and reasoning always belonged to the displayed frame. We addressed this through frame identifiers, bounded caches, and explicit handling of stale or missing data.

Geographic information required careful treatment. An aircraft’s GPS position does not establish a survivor’s location. We kept missing detection locations explicit and treated camera-footprint estimates as approximate information.

We also encountered an AWS account policy that blocked Kinesis stream creation. We retained the Kinesis integration plan, and the cloud demonstration reads the public feed directly.

Accomplishments that we're proud of

Connected video ingestion, perception, spatial reasoning, action decisions, and operator visibility into one working system.

Delivered a dashboard showing target, confidence, hazard, location, and current state alongside synchronized video.

Implemented human-review escalation, commander override controls, and a bounded decision history that operators can inspect.

Built reproducible mock workflows and automated tests covering contracts, streaming, reasoning, authentication, and action handling.

Documented a held-out evaluation across 1,264 sampled frames from DJI_0804, which was excluded from training. At confidence 0.45 and IoU 0.50, the trained boat detector achieved 87.0% precision and 90.2% recall. Person recall on the same frames was 20.1%, so small and distant swimmers remain the priority.

What we learned

We learned that useful rescue automation requires more than object detection. A detection must carry trustworthy confidence, location context, frame identity, and a clear explanation of the resulting decision.

We also learned to distinguish model validation results, held-out evaluation, demonstration behavior, and deployment measurements. Each answers a different question. Missing coordinates, uncertain detections, and stale footage must remain visible to operators.

Working across five engineering roles reinforced the value of shared schemas, clear interfaces, reproducible examples, and integration tests.

What's next for Aegis

Our next priorities are to:

Improve detection of small and distant survivors using more varied training data and targeted evaluation.
Strengthen geographic projection and synchronize camera telemetry more accurately with video.
Connect action intents to vehicle simulators, then evaluate supervised hardware integration.
Add persistent audit storage and strengthen deployment security and reliability.
Resolve the Kinesis account restrictions and evaluate the full streaming path.
Run measured Graviton and COOL evaluations, including throughput, latency, and compute cost.

Graviton, COOL, and compute-cost results remain future work.

Built with

Python, OpenCV 5, NumPy, Ultralytics YOLO11n, PyTorch, FastAPI, Pydantic v2, Uvicorn, asyncio, WebSockets, MCP, JSON-RPC, Streamlit, HTML, CSS, JavaScript, Docker, Amazon ECR, Amazon ECS Fargate, boto3, pytest, and Node.js.

Datasets: MOBDrone and SeaDronesSee.