Watcher: AI-Powered Situational Awareness for First Responders
Inspiration
In an emergency, every second matters. But the people making life-or-death decisions are often working with incomplete information.
Picture a firefighter entering a burning building. Over the radio, responders are reporting injuries, identifying hazards, requesting backup, and coordinating evacuations. Meanwhile, the incident commander is trying to maintain a mental picture of everything happening: who's injured, where they're located, which units are responding, and whether everyone is safe.
The problem isn't a lack of information. It's that critical information is scattered across conversations, radio transmissions, and individual responders.
Junior responders face an additional challenge. During their first major incident, they may know their training but struggle to recall the right next step under pressure. And when a responder falls or loses consciousness, valuable seconds can pass before anyone realizes something is wrong.
At VexoHacks, we were introduced to a wearable wristband equipped with a microphone and motion sensors. That sparked a question:
What if first responders could focus on saving lives while AI automatically built the incident picture around them?
That's why we built Watcher, a real-time, multimodal AI incident intelligence platform that turns voice, video, and wearable sensor data into actionable situational awareness.
What It Does
Watcher transforms chaotic emergency communications into a shared, continuously updated operational picture.
Instead of requiring responders to enter information manually, Watcher listens, observes, organizes, and communicates.
1. Voice-to-Incident Intelligence
A responder says:
"Two victims on floor three. One adult isn't breathing, and one child is coughing."
Within approximately 1 to 1.5 seconds, Watcher identifies the patients, extracts their locations and reported conditions, and creates structured incident records.
The dashboard now displays two casualties with suggested triage priorities: RED and YELLOW.
But emergencies evolve. When someone says, "Medic 1 is taking the red patient," Watcher updates the existing record rather than creating a duplicate.
If another responder reports, "Correction, the child isn't breathing either," the system updates the child's condition and escalates the suggested priority to RED.
Watcher doesn't just transcribe conversations. It understands how an incident changes over time.
2. Multimodal Awareness Through Camera Vision
Voice reports tell only part of the story.
Watcher combines spoken observations with visual context from helmet-camera footage or a live phone-camera stream. A vision agent analyzes relevant frames to provide additional descriptions of casualties and surrounding conditions.
This gives incident commanders another source of information without requiring responders to manually document everything they encounter.
3. AI Coaching for Junior Responders
During an emergency, junior responders may need quick access to relevant procedures.
Watcher provides concise, spoken, protocol-based guidance and supports hands-free questions.
For example, a responder can ask about immediate considerations when encountering an unresponsive patient.
The AI serves as a decision-support assistant, not a replacement for trained responders, established protocols, or incident command.
4. Automatic Responder Distress Detection
One of Watcher's most important features is its integration with the Vexo wearable's motion sensors.
When the wristband detects a potential fall followed by sustained stillness, Watcher initiates an automated safety sequence:
- Detect a potential fall.
- Allow one second for the landing motion.
- Confirm five seconds of stillness.
- Start a ten-second cancellation countdown.
- Let the responder double-tap the wristband to confirm they're safe.
- If the countdown expires, issue an automated spoken mayday containing the responder's last recorded words.
The prototype's alert timing is deliberately deterministic:
$$ t_{\text{mayday}} = t_{\text{fall}} + 1\text{s} + 5\text{s} + 10\text{s} $$
$$ \boxed{t_{\text{mayday}} = t_{\text{fall}} + 16\text{s}} $$
This is the designed timing for a qualifying fall-and-stillness event, not a guarantee of detecting every emergency.
5. Automatic After-Action Reporting
When the incident ends, the documentation is already taking shape.
Watcher generates an after-action report containing:
- A chronological incident timeline
- Patient and triage summaries
- Resource requests and unresolved needs
- Responder fall events and subsequent actions
- Recorded response times and operational milestones
Instead of reconstructing an incident from memory and fragmented radio logs, commanders can review an organized event history.
From the first radio transmission to the final report, Watcher turns communication into coordinated intelligence.
How We Built It
We designed Watcher as an event-driven, multi-agent system that connects wearable hardware, real-time speech recognition, computer vision, AI reasoning, and a live operational dashboard.
Hardware and Real-Time Audio
The Vexo wristband streams Opus audio and motion events over Bluetooth to a Python bridge running locally.
We use Deepgram for streaming speech-to-text, allowing spoken reports to be processed as responders communicate.
The same bridge handles motion events using a deterministic fall-detection state machine:
WATCHING → CHECKING → COUNTDOWN → MAYDAY
Five Specialized AI Agents
We built our agent orchestration using Mastra, with five agents responsible for different aspects of incident management:
| Agent | Technology | Responsibility |
|---|---|---|
| Intake | Gemini 3.1 Flash-Lite | Converts speech into patients, hazards, updates, and resource requests |
| Vision | Gemini 3.8 Flash | Analyzes camera frames for additional visual context |
| Coach | Gemini Flash-Lite | Produces concise, protocol-based guidance |
| Comms | MiniMax-M3.1-Flash | Creates hospital pre-alert communications |
| Analyst | MiniMax-M3 | Generates narrative after-action analysis through a Mastra workflow |
OpenAI models provide a fallback reasoning path across the agents, strengthening the system's resilience when a primary model is unavailable.
MiniMax Speech powers spoken coaching and alerts, with additional text-to-speech fallback options.
Deterministic State Management
One of our most important architectural decisions was separating AI interpretation from application state.
The AI proposes. Deterministic code decides.
Agents return structured operations rather than directly modifying the incident dashboard.
A deterministic reducer validates and applies those operations, enforcing patient identity, update rules, and state consistency.
This helps prevent duplicate patients, invalid references, and inconsistent incident records.
Similarly, response-time calculations and report metrics are computed directly from recorded events. The Analyst agent generates the narrative but cannot independently change those numbers.
Live Incident Dashboard
A lightweight event server distributes updates through Server-Sent Events to the browser and connected agents.
As reports arrive, the dashboard updates automatically, providing shared visibility into patients, incidents, hazards, resources, and responder status.
Deployment and Camera Integration
We deployed Watcher's dashboard and agent services to Superserve, making the application accessible through a public URL.
The Bluetooth hardware bridge remains on the local laptop, while remote viewers can follow the incident in real time.
We also implemented read-only viewer access and operator-key-controlled actions.
For visual input, we support helmet-camera footage and live phone-camera streaming with QR-code pairing, allowing responders to contribute visual context without specialized streaming hardware.
Challenges We Ran Into
Building a real-time AI application is challenging. Building one that integrates physical hardware, live audio, video, multiple models, and cloud infrastructure introduces an entirely different level of complexity.
1. Our AI Started Listening to Itself
When Watcher spoke a coaching instruction through the laptop speakers, the wristband microphone picked it up.
Deepgram then transcribed the AI's own words as if a responder had said them.
We implemented an echo guard that compares incoming transcripts against recent system-generated speech.
Our initial implementation was overly aggressive. It removed a legitimate phrase, "on floor 3," because the coaching system had recently said "on floor three."
We refined the guard to require at least four consecutive matching words before filtering potential echoes.
2. The Microphone Appeared Connected Even When Transcription Failed
Deepgram occasionally disconnected while the Bluetooth wristband remained connected.
From the operator's perspective, everything appeared normal, even though no speech was being processed.
We added automatic reconnection and explicit live transcription indicators.
A connected device doesn't necessarily mean a functioning data pipeline.
3. Hardware Didn't Behave Like the Documentation
The provided GoPro couldn't support the live-streaming workflow we initially envisioned.
Rather than abandon visual intelligence, we built a phone-camera streaming alternative with QR-code pairing.
We also discovered that the wristband supported only one Bluetooth connection at a time. A teammate's USB-connected session unexpectedly took control of one of its streams.
These issues reinforced the importance of testing with actual hardware rather than relying on simulated inputs.
4. Cloud Deployment Exposed Unexpected Failures
Our camera footage used AV1 encoding, which the cloud sandbox couldn't decode. We converted the video pipeline to H.264.
We also encountered an HTTP connection-reuse issue where rejecting a request without consuming its body interfered with the next request.
Finally, our tightly coupled video synchronization created stuttering whenever network latency increased. We adjusted the synchronization strategy to tolerate delays more gracefully.
5. Getting Latency Low Enough for Real-Time Use
Our initial intake pipeline took approximately three seconds to update the incident dashboard.
For emergency response, that delay felt significant.
After benchmarking models, we moved intake processing to Gemini Flash-Lite and reduced updates to approximately 1 to 1.5 seconds, while maintaining the same results across our accuracy test cases.
What We Learned
1. Reliable Agentic AI Requires Guardrails
Our biggest engineering lesson was that LLMs are excellent at interpreting unstructured information, but they shouldn't independently control critical application state.
Separating AI-generated proposals from deterministic state transitions made Watcher easier to debug, test, and reason about.
2. Multimodal AI Is More Powerful Than Any Single Input
Speech communicates observations. Video provides visual context. Motion sensors detect physical events.
Combining these signals creates a much richer incident picture than transcription alone.
3. Graceful Degradation Matters
External services can disconnect, slow down, or become temporarily unavailable.
We implemented fallback paths for reasoning models and voice synthesis, including OpenAI for model inference and MiniMax, ElevenLabs, and Gemini TTS for speech generation.
4. Real-World Testing Reveals Problems Simulations Miss
Echo, background conversations, Bluetooth contention, video encoding, and dropped connections became visible only when we tested Watcher with actual hardware and people speaking in the same room.
5. Automated Testing Is Essential
We built an end-to-end, nine-step incident simulation against the hosted application.
The scenario passed 15 out of 15 validation checks, supported by more than 60 additional unit and live-model tests.
These results validate the prototype workflows, although real-world emergency use would require substantially more testing and professional evaluation.
What's Next
Watcher is a hackathon prototype, but our long-term vision is a connected intelligence layer for emergency response teams.
Our next priorities include:
- Fully conversational voice: Integrate ElevenLabs to support natural, multilingual, hands-free conversations between responders and Watcher.
- Multi-responder coordination: Connect multiple wristbands and cameras to one shared incident board, with separate responder identities and synchronized event histories.
- Resource optimization: Use incident intelligence to recommend resource allocation, hospital coordination, and response-route adjustments.
- Incident cost intelligence: Integrate Brex to help track operational expenditures and resource-related costs.
- Accessible deployment: Explore Stripe-based per-responder pricing so smaller and volunteer departments can adopt the technology.
- Field validation: Work with firefighters, EMS personnel, and incident commanders to evaluate accuracy, alert reliability, safety, and usability before any operational deployment.
Our Vision
Today, emergency responders are expected to communicate what they see, remember what others have said, coordinate resources, protect themselves, and document everything afterward.
We believe technology should carry more of that information burden.
Watcher is our first step toward an emergency response system that listens when responders speak, understands when situations change, recognizes when someone may need help, and keeps everyone working from the same picture.
Because in an emergency, responders should spend less time managing information and more time saving lives.
