OmniWatch: Real-Time AI Emergency Detection Platform

• 

  Thumbnail

Inspiration

Every year, people die in emergencies not because help wasn't available — but because of delay. A security guard watching 12 feeds simultaneously misses the one that matters. Someone collapses in a lobby at 9 PM and no one notices for minutes. In cardiac arrest, survival probability drops roughly 10% every minute without intervention. We built OmniWatch because we believed that gap — between incident and response — could be closed with AI.

What it does

OmniWatch is a real-time AI safety system that monitors live camera feeds and audio streams to detect emergencies, classify their severity, and automatically route the optimal medical response — all within 10 seconds.

• Vision AI (YOLOv8): Detects falls, fights, and motionless persons using optical flow energy analysis across bounding boxes
• Audio AI (Whisper + librosa): Identifies screams, distress calls, and impact sounds in 64ms audio windows
• Agentic AI (Claude): Fuses both signals into a unified confidence score, classifies severity (CLEAR → CRITICAL), reasons over real SF hospital data, and dispatches the optimal medical response with a plain-English incident report
• Smart Routing (OSRM): Selects the optimal hospital by minimizing a weighted score over ETA, trauma level, and distance — using real road geometry, no paid APIs

The system targets sub-10-second detection — a 30× improvement over the traditional human response chain — and up to a +41 percentage-point improvement in survival probability for cardiac events.

How we built it

• Frontend: Vanilla JS dashboard with Leaflet.js + CartoDB Dark Matter tiles, animated ambulance routing via OSRM, real-time WebSocket updates, and a full dark liquid glass morphism UI
• Backend: FastAPI with async WebSocket pipeline serving detection results in real time
• Vision pipeline: YOLOv8 for person detection and bounding boxes, OpenCV for optical flow energy computation across frames
• Audio pipeline: Whisper for speech recognition and distress keyword matching, librosa for short-time energy waveform analysis
• Agentic layer: Claude API for multi-factor reasoning — fusing $C_{fused} = 0.65 \cdot C_{vision} + 0.35 \cdot C_{audio}$, classifying severity, and selecting optimal dispatch via $h^* = \arg\min [w_1 \cdot \text{ETA}(h) + w_2 \cdot \frac{1}{\text{TraumaLevel}(h)} + w_3 \cdot d(h)]$
• Desktop app: Packaged as a native Mac application using Electron, serving the frontend over localhost so WebAuthn biometric auth works natively

Accomplishments that we're proud of

• A genuinely beautiful, production-quality security operations dashboard that judges can interact with live
• Real ambulance routing that follows actual San Francisco roads, animating in real time with a trailing solid line and dashed remaining route
• The math actually works: our survival probability model, confidence fusion formula, and hospital routing optimization are grounded in real emergency medicine literature
• Shipping a working Mac desktop application from a web frontend in under a day using Electron
• A fully documented, clean architecture with no paid API dependencies

What we learned

• Claude is exceptionally well-suited for agentic reasoning tasks that require balancing multiple competing factors simultaneously — the hospital routing decision is genuinely a multi-objective optimization problem, and framing it as a reasoning task for Claude produces better results than a hardcoded scoring function
• WebAuthn is powerful but unforgiving about origins — a single wrong URL scheme breaks the entire biometric flow
• Special characters in database passwords (@) silently break SQLAlchemy connection URLs unless URL-encoded (%40)
• Open-source mapping (Leaflet + OSRM + CartoDB) is genuinely production-ready and visually indistinguishable from paid alternatives when styled correctly

What's next for OmniWatch: Real-Time AI Emergency Detection Platform

• Direct 911 CAD integration: Automatic dispatch handoff when $C_{fused} \geq 0.97$
• Multi-camera Kalman tracking: Following an incident across multiple feeds simultaneously
• Wearable integration: Smartwatch accelerometer data feeding into the optical flow energy model
• Crowd density risk detection: Flagging dangerous density levels before incidents occur
• Cloud deployment: AWS EC2 backend with HTTPS for full WebAuthn biometric support outside localhost
• Online learning: Updating detection thresholds from confirmed vs. false positive feedback loops

GitHub: https://github.com/ssyht/Claude-Hackathon-OmniWatch

Built With

• agentic-ai
• anthropicapi
• computer-vision
• docker
• javascript
• mysql
• whisperai
• yolo