AEGIS

The camera feed from this DJI Matrice is processed by our n8n automation to speed up emergency response.
Our dashboard live streams video footage from the drone, and the view can be captured for analysis by our workflow.
A n8n automation enables GPT-OSS-120B to access the USDOT Emergency Response Guidebook to assist first responders in an emergency response.
After the n8n automation, a first responder receives an answer in seconds, instead of having to manually go through a 400 page book.

Introduction: Solving a Big Problem for First Responders

Having spent considerable time developing technology for first responders, I've seen firsthand the immense challenges they face and the profound gap between the tools at their disposal and the technology we often take for granted. The field is ripe for innovation, but the solutions must be robust, reliable, and accessible.

When I saw the release of GPT-OSS-120B, it wasn't just another impressive model; it was a catalyst. I immediately recognized a set of unique characteristics that made it the perfect keystone for a new generation of emergency response systems:

Accessibility and Affordability: The cost-efficiency of the model, relative to a hypothetical GPT-5, makes it a viable tool for public safety agencies operating on tight budgets. Furthermore, the existence of smaller variants, like a potential GPT-OSS-20B that could run on a high-end gaming laptop, solves one of the biggest challenges in the field: operation in disconnected environments where internet access is a luxury, not a guarantee.

Agentic and Function-Calling Capabilities: This was the true "aha!" moment. For the first time, a powerful open-source model wasn't just a language processor; it was an engine for action. The ability to not only understand natural language but to reliably reference vast repositories of specialized documentation—the very protocols first responders live by—and then trigger programmatic actions based on that data, was a complete game-changer.

That's when I had an idea. What if we could take this powerful, affordable, and actionable AI and integrate it directly with the robotic systems already being deployed in the field? What if we could close the loop between seeing, understanding, and acting?

This brings us to the problem AEGIS was born to solve.

Problem: Outdated Protocols for Emergency Response

First responders are held back by technology that has remained largely unchanged for decades. This gap is never more apparent, or more dangerous, than during a hazardous materials (HAZMAT) incident.

Imagine a tanker truck jackknifed on a highway at 2 AM. The first unit arrives on scene. The commander’s immediate priorities are the safety of their crew and the public. They see a diamond-shaped placard on the side of the truck with a 4-digit number, but it's too far away to read clearly. They deploy a drone to get a closer look.

The drone's camera identifies the number: 1017.

This is where the high-tech, 21st-century response grinds to a halt. The next step is a manual, analog process fraught with potential for error under extreme stress. An officer must physically retrieve a 400-page book from their vehicle—the Emergency Response Guidebook (ERG), published by the U.S. Department of Transportation. Standing in the dark, potentially in the rain, with flashing lights creating a chaotic scene, they must:

1) Manually Look Up the ID: Flip through the yellow-bordered pages, scanning hundreds of entries to find the number 1017.

2) Find the Cross-Reference: The entry for 1017 tells them this material corresponds to Guide 124.

3) Manually Look Up the Guide: They then flip to a completely different, orange-bordered section of the book to find Guide 124.

4) Verbally Relay Information: The commander reads the critical information from the page—hazards, required protective gear, initial isolation distances—and verbally relays these complex instructions over the radio to dispatch and other responding units.

This entire process can take several minutes. In an incident where a toxic cloud could be spreading, minutes are an eternity. It is a system that relies on a human's ability to perform a perfect, high-speed lookup in a dense reference manual under the worst possible conditions.

The Solution: AEGIS

Aegis is the answer to this challenge. It is an Autonomous Emergency Guidance & Intervention System that closes the loop between robotic perception and intelligent, automated action. Our solution is not just a digital book; it is a complete, end-to-end pipeline that transforms raw visual data into a life-saving response in seconds. This is how we address the judging criteria:

Application of gpt-oss & Novelty of the Idea The novelty of Aegis lies in its architecture, which uses GPT-OSS-120B not as a simple text generator, but as the central reasoning engine in a real-time operational loop. The workflow is a paradigm shift from current technology:

Perception (VLM): A drone-mounted camera sends an image to a Vision Language Model. This model’s only job is to perform a targeted OCR scan, extracting the 4-digit UN ID from the HAZMAT placard.

Comprehension (RAG): The extracted ID (1017) is used to perform a hyper-accurate, metadata-filtered query on our specialized Pinecone vector databases. This retrieves the exact Guide ID ("124") and then the full text of that one specific guide.

Reasoning & Action (GPT-OSS-120B): This is where the unique strength of a large, open model shines. The perfect, isolated context of Guide 124 is fed to GPT-OSS-120B. It synthesizes this dense text into a clear, actionable plan and—most importantly—uses its function-calling capabilities to format this plan into a machine-readable JSON object. This structured output is what allows us to trigger automated workflows, a task far beyond the scope of simple text generation. Aegis turns a language model into an autonomous incident coordinator.

Design & Potential Impact Recognizing that the end-user is a first responder under immense stress, our design philosophy is built around a "Cognitive Co-Pilot" dashboard.

The Interface: The UI is a clean, two-panel layout: the live drone feed on the left, and the AI's analysis on the right. The dark theme provides high contrast for legibility in chaotic lighting conditions. When the AI completes its analysis, it presents a "Digital ERG Card" with clear, icon-driven information: hazards, PPE requirements, and a map with safety zones.

Human-in-the-Loop Safety: The design is fundamentally safe. The AI presents its findings and suggests recommended actions (e.g., "Request HAZMAT Team," "Initiate Evacuation Notice") as buttons. Crucially, these actions require explicit human confirmation, ensuring the incident commander always has final authority. This demonstrates a thoughtful and balanced blend of backend automation and frontend safety.

The potential impact is immense. For the first responder, Aegis turns a multi-minute manual process into a reliable, sub-five-second automated workflow, directly contributing to faster and safer incident outcomes. Beyond HAZMAT, this architecture serves as a foundational blueprint for a new generation of intelligent field robotics. The same system can be adapted for search and rescue, wildfire assessment, or industrial inspection, democratizing expert knowledge and creating a safer, more efficient world.

How we built it

Aegis is a full-stack application built for rapid, reliable deployment. The frontend is a Next.js dashboard using shadcn/ui for a clean, responsive interface. A Python Flask server captures video frames from the drone feed. The core AI pipeline is orchestrated in n8n, connecting a VLM for OCR, two specialized Pinecone vector stores for hyper-accurate ERG lookups, and GPT-OSS-120B for the final reasoning and function-calling.

Accomplishments that we're proud of

We are most proud of developing a highly specialized, two-step RAG pipeline for the ERG, which provides near-100% accuracy for a life-saving use case. This system is a fully integrated, end-to-end solution, from drone video capture to a safety-focused UI, that successfully closes the loop from perception to autonomous, human-confirmed action.

Our software can also interact with numerous other UAV and robotic platforms, such as Skydio or Parrot. Anything that has a camera can be made to stream to our application, which offers tons of customizability and versatility.

What's next for AEGIS

This hackathon project is the beginning of a real-world solution. We presented our HAZMAT demo to a Deputy State Fire Marshal who specializes in these incidents in Utah. His excitement was palpable, and we are now actively collaborating to develop Aegis further for the specific needs of first responders.

The HAZMAT protocol is just the first step. Our vision is to build out a comprehensive library of automated protocols—from mass casualty incident (MCI) triage to wildfire assessment and search and rescue patterns—that can be natively integrated with modern robotics platforms. This ambition, however, hinges on access to AI that is not only powerful but also affordable, effective, and capable of being run locally in disconnected environments.

This is why OpenAI's open-weight models are so transformative. They represent the key to offering this functionality at scale, providing the agentic capabilities needed to turn every drone into an expert co-pilot and every first responder into a more effective, better-protected hero

Thank You

We are very grateful for the opportunity to compete in this hackathon. Thank you to everyone over at OpenAI, NVIDIA, HuggingFace, Ollama, vLLM, LMStudio, and of course Devpost for this wonderful experience!