Sentinel

Inspiration

We were inspired by the unfortunate reality that personal safety is a growing concern for everyone, especially in urban environments. We noticed that traditional safety apps are reactive; they require you to press a button when something is already wrong. But what if you can't reach your phone? What if you're in a situation where speaking is dangerous? We wanted to build a "guardian angel" that is proactive, context-aware, and can act on your behalf when you cannot. We wanted to bridge the gap between technology and physical security in a way that feels human and reassuring, not cold and robotic.

What it does

Sentinel is a comprehensive AI-powered personal safety system. It turns your smartphone into an intelligent protector. key features include:

AI 911 Proxy: If you are in danger and cannot speak, Sentinel uses AI to call emergency services for you. It listens to the operator and responds independently, describing your live location, the situation, and even visual details from your camera feed using computer vision.
Dead Man's Switch: A proactive check-in system. If you don't respond to safety checks while walking home alone, Sentinel automatically alerts your contacts and starts streaming evidence.
Live Emergency Streaming: Instantly broadcasts video and audio to your trusted contacts, saving the evidence securely to the cloud even if your phone is destroyed.
Fake Call: A realistic tool to help users exit uncomfortable social situations.
Offline Resilience: It works when the internet doesn't, queuing alerts and data to sync the moment connection is restored.

How we built it

Real-Time Communications & Signaling Architecture:

Event-driven Node.js/Express backend utilizing Socket.IO for low-latency bi-directional WebSocket streams to manage instantaneous state synchronization across clients.
Integrated Twilio Programmable Voice SDK via TwiML for autonomous telephony bridging, enabling the AI to interface directly with PSTN emergency networks.
Custom stream chunking & buffering algorithms designed to manage real-time audio/video data flow between client, server, and AI models with minimal latency.
WebRTC principles applied for managing media streams efficiently across varying network conditions.

Multimodal AI Analysis Pipeline:

Google Gemini 1.5 Flash integration via REST API for frame-by-frame visual reasoning and situational context extraction, analyzing video feeds in real-time.
TwelveLabs Marengo-3.0 video understanding engine for semantic search and granular activity recognition, allowing deep post-event analysis of recorded evidence.
ElevenLabs Turbo v2.5 TTS engine optimized for sub-500ms latency voice synthesis, creating human-like responses vital for maintaining engagement with emergency operators.
Browser-side Web Speech API for robust, privacy-first wake word detection ("Help me Sentinel", "Help," "Emergency," & "Sentinel") and speech-to-text processing without continuous cloud dependence.

Offline-First & PWA Foundation:

Vite and React powered frontend utilizing Service Workers with Stale-while-revalidate caching strategies for instant load times and offline functionality.
IndexedDB implementation (via localforage) for persistent local storage of emergency recordings and pending sync queues, ensuring data integrity during connectivity loss.
Background Sync API utilization to ensure critical alerts are dispatched immediately upon network restoration.

Sensor Fusion & Hardware Integration:

Real-time Geolocation API tracking with high-accuracy fallbacks and battery-efficient monitoring.
DeviceMotion API utilization for shake detection usage accelerometer heuristics to trigger SOS modes without UI interaction.
MediaStream API orchestration for seamless camera/microphone access management handling diverse mobile browser permissions and constraints.

Challenges we ran into

The biggest challenge was handling latency in the "AI 911 Proxy" feature. Connecting a live phone call (via Twilio) to an AI brain (Gemini/OpenAI) and then generating speech (ElevenLabs) takes time. We spent a LOT of time optimizing the pipeline, parallelizing the transcription and thinking processes to get the response time down to a conversational level.

Accomplishments that we're proud of

We are incredibly proud of the AI Proxy. Watching the system successfully hold a conversation with a real phone line, describing a video feed it was "seeing" in real-time, was probably the highlight of our hackathon. We're also proud of the offline-first architecture; knowing that our app can still save a life even with spotty cellular service is a huge technical and ethical win for us.

What we learned

We learned that simplicity is a feature. In a panic, users have zero cognitive load to spare. We had to cut down steps and simplify the UI. We also learned a ton about the intricacies of modern browser permissions; getting microphone, camera, and geolocation to play nicely together in the background required deep dives into browser privacy protocols.

What's next for Sentinel

We plan to introduce "Sentinel Wear," a companion app for smartwatches to trigger alerts based on biometric data like sudden heart rate spikes. We also want to expand our computer vision capabilities to automatically detect specific weapons or aggressive body language to trigger alerts proactively before a situation escalates.