Guardian

Inspiration

Emergency response has a coordination problem. When a fire breaks out, a crash scatters pedestrians, or a fight erupts in a public space: the seconds between detection and dispatch are the difference between life and death. But today, most emergency systems still rely on someone manually calling 911, waiting on hold, and hoping the right responders get there in time.

We asked: what if the system saw the threat before anyone made a call?

Guardian was built to answer that question. We wanted to create a command center that watches, thinks, and acts (automatically) so first responders can focus on what they do best.

The Pipeline

Challenges We Ran Into

1. False Negatives on Partial-Frame Events

Early versions sampled at 320×180 every 8 seconds with detail: "low". A vehicle crash went undetected. We switched to 640×360, 3-second intervals, detail: "high", and rewrote the system prompt to explicitly tell the model that corner-of-frame events count. Detection rate improved dramatically.

2. Duplicate Incident Creation

DB refreshes after incident creation caused camera components to re-render with new props, resetting dedup refs and re-triggering analysis. Fixed by keying the analysis useEffect only on [localVideoUrl, feed.id].

3. Real-Time Map Accuracy

A generic SVG map produced wildly wrong pin locations. We switched to react-simple-maps with geoAlbersUsa projection and a CITY_COORDS lookup for 50 US cities.

Key Features

Sponsor Tracks

Thanks to the many sponsors that made this project possible. Here's how we've used them.

Best Use of Responsible AI

In developing this project, we needed to acknowledge the limitations of AI in the usage of our product. We wanted there to be a built-in fail safe in the instance that there wasn't a clear classification for whether there was an emergency risk present in the camera footage. In that case, the incident request is flagged for manual review, and operators managing Guardian can review that request, deciding whether to proceed or discard it. Here, we prioritize the value in human review.

Best Use of ElevenLabs

We integrated a voice agent from ElevenLabs to act as our source contact bridging the incident reports and the appropriate authorities. It takes the information from the incident report created, and communicates that information to the authorities; it's capable to answer any questions based on the information provided to it, and only the information provided to it. We wanted to place guardrails on the communication of detailed information that's trustworthy and vetted based on camera footage.

Best Use of WebSpatial

We’re using WebSpatial to generate a live map visualization of calls coming through our incident reporting system, giving teams a clear geographic view of where activity is happening and how situations are unfolding. From that map, users can drill down into specific events and, when available, view clips of the actual incident in visionOS for a more immersive perspective. The goal is to move beyond static reports and give responders and decision-makers better situational awareness, faster context, and a clearer understanding of what’s happening on the ground so they can act with more confidence.

Best Use of AMD

We prototyped a locally-trained hazard detection model on AMD Developer Cloud (MI300X, ROCm + PyTorch), fine-tuning VideoMAE on UCF-Crime across 1,610 clips. After two training rounds with class-weighted loss, we hit 62% accuracy, but only 5 of our 11 hazard classes had any training data at all. With flood, hazmat, structural, medical, and environmental having zero representation, we're staying on GPT-4o for now. Once we have a complete labeled dataset, AMD's infrastructure gives us a fast, cost-effective path to a fully self-hosted pipeline.

What's Next for Guardian

• Live RTSP/WebRTC camera stream support
• SMS + push notification fallback
• Historical analytics and incident heatmaps
• Role-based access control
• Mobile field app

Built With

• computer-vision
• elevenlabs
• typescript
• webspatial