RignSentinelAI

RingSentinel AI links smart home cameras with public safety alerts. Its multi-agent AI visually verifies real-time emergency threats & routes encrypted dispatch packages directly to emergency services

Comment

Inspiration

Emergency broadcasts—such as Wireless Emergency Alerts (WEA) for AMBER alerts, active threats, and silver alerts—often reach communities minutes or hours after a critical incident begins. Meanwhile, thousands of residential smart doorbells and security cameras capture continuous high-resolution feeds but remain isolated silos without situational context. Inspired by predictive infrastructure triage principles, RingSentinel AI bridges this gap by transforming passive perimeter cameras into an active, event-driven intelligence network that automatically cross-references emergency dispatch vectors with edge camera feeds in real time.

What it does

RingSentinel AI ingests raw public safety alert payloads, dynamically constructs threat vectors, evaluates visual perimeter feeds via Ring camera hardware, and orchestrates automated emergency dispatch packages:

Ring SDK & Device Auth: Authenticates securely with Ring account endpoints using 2FA/token-refresh flows to query device manifests (Doorbots, StickUp Cams, Spotlight Cams) and pull live motion snapshots (async_get_snapshot()).

Dual-Mode Ingestion: Operates via direct synchronous HTTP for low-latency testing or asynchronous AWS SQS long-polling queues for handling regional alert spikes.

Spatial-Temporal Deduplication: Employs Redis sliding-window caching (dedupe:wea:*) to suppress duplicate alert processing within configurable sliding windows.

Dynamic Target Vectoring: Extracts threat categories, physical descriptors, and spatial keywords using structured LLM outputs.

Multimodal Edge Analysis: Leverages Anthropic Claude 3.5 Sonnet on Amazon Bedrock alongside Model Context Protocol (MCP) tool bindings (ring-mcp-server) to evaluate Ring motion frames against physical threat descriptors.

Adaptive Risk Engine: Adjusts confidence thresholds dynamically based on ambient conditions (e.g., lower thresholds during nighttime IR sensor noise) to determine operational escalation.

Encrypted CAD Archival: Generates standardized CAD packages, encrypts evidence metadata via AWS KMS, and archives immutable records directly to an S3 Evidence Vault.

How we built it

Application Framework: Built with Python 3.12 and FastAPI using uv for dependency resolution, pydantic for strict payload validation, and structlog for OpenTelemetry correlation tracing. Ring Ecosystem & SDK Integration: Integrated ring_doorbell Python SDK via an asynchronous wrapper client. Implemented OAuth2 token persistence (/tmp/ring_token.json / Secret Store injection) and Session Management (async_create_session()) with automatic 2FA token refresh hooks (token_updated) to safely stream frames from doorbells and cameras without exposing plain credentials.

Agentic Graph Workflow: Powered by LangGraph to execute stateful state transitions across specialized agents (AlertScout $\rightarrow$ Supervisor $\rightarrow$ PerimeterSentinel $\rightarrow$ SafetyGuard).

Tool & Model Integration: Uses FastMCP for decoupled server tools (ring-mcp-server for Ring device camera snapshot extraction, bedrock-mcp-server for Claude 3.5 Sonnet multimodal frame reasoning, and cad-mcp-server for automated law enforcement dispatch package generation) along with LLMFactory supporting both local LM Studio (Qwen 3.5) and cloud-hosted AWS Bedrock models.

Infrastructure as Code (IaC): Fully provisioned on AWS using modular Terraform, deploying AWS App Runner (linux/amd64), ElastiCache Valkey (cache.t4g.micro), Amazon ECR, Amazon SQS/DLQ, KMS CMK, S3 Evidence Vault, and GitHub Actions OIDC identity federation.

Challenges we ran into

Agentic Graph Cycles & Infinite Loop Mitigation: Designing the LangGraph supervisor routing logic required precise conditional guardrails. When visual confidence scores fell into ambiguous mid-ranges, the supervisor risked oscillating between AlertScout parsing and PerimeterSentinel frame verification, requiring strict recursion limits and adaptive confidence state checks to force deterministic state convergence.

Model Context Protocol (MCP) Tool Call Reliability: Integrating fastMCP servers (ring-mcp-server, bedrock-mcp-server, cad-mcp-server) presented serialization and transport challenges. Handshaking tool schemas cleanly between local runtimes and Bedrock VLM endpoints required strict input validation and fallback mechanisms when mock frames or live SDK connections timed out.

VLM Visual Reasoning Latency vs. Real-Time Ingress: Multimodal reasoning via Anthropic Claude 3.5 Sonnet on Bedrock can introduce non-negligible processing latency. Balancing real-time alert ingestion with high-accuracy frame analysis required offloading heavy VLM tool calls into non-blocking asynchronous graph nodes while keeping the FastAPI/SQS ingress thread completely responsive.

Stateful Checkpointing & Memory Bloat: Managing state persistence across LangGraph execution cycles using Redis required careful schema design. Storing full base64 snapshot frames directly in state caused memory overhead, prompting us to separate frame metadata from state payloads and prune stale checkpoint keys via sliding TTL windows.

Deterministic Output Enforcement from Non-Deterministic LLMs: Converting unstructured public safety alerts into structured TargetVector objects required strict Pydantic model enforcement. Handling edge cases where LLMs returned partial or malformed extraction schemas required robust retry logic and default state fallback handling.

Accomplishments that we're proud of

Seamless Ring Hardware Integration: Successfully connected the ring_doorbell SDK into a decoupled FastMCP tool framework, enabling dynamic frame extraction across doorbots and camera streams.

Zero-Downtime Eventing Engine: Built a resilient background SQS polling daemon with exponential backoff, dead-letter routing, and Prometheus metrics tracking.

Sub-Dollar Fixed Baseline Cost: Engineered a cost-effective cloud architecture running on App Runner and micro Valkey caching nodes for under $30/month.

End-to-End Encryption & Security: Enforced KMS server-side encryption for S3 evidence vaults, isolated private VPC endpoints, and password-stdin ECR authentication.

Modular Multi-Agent Pipeline: Successfully decoupled vision reasoning, threat vectoring, and CAD dispatching into isolated LangGraph nodes with full state persistence.

What we learned

Ring Ecosystem APIs: Understanding token lifecycle management, 2FA authorization flows, and async session handling across Ring Doorbots and StickUp Cams.

MCP Governance: How Model Context Protocol standardizes multi-agent tool execution across local and cloud environments.

Adaptive Thresholding as Circuit Breakers: Static confidence scores fail when visual sensors operate in noisy environmental conditions (e.g., nighttime IR feed noise or motion blur). Implementing dynamic confidence evaluation inside edge routing logic prevented premature agent escalations while ensuring high-risk edge cases triggered mandatory visual verification.

Fault Isolation & Graceful Fallbacks: Isolating hardware and external API calls behind MCP tool contracts protected the main agent state from crashing when live camera streams or external auth tokens timed out, allowing the pipeline to fallback gracefully to mock feeds or retry queues.

Pydantic Model Enforcement: Translating unstructured, noisy public safety alerts into structured downstream actions requires strict runtime validation schemas (TargetVector). Enforcing structural output models at the agent boundary prevents malformed LLM outputs from propagating errors into law enforcement dispatch workflows.

What's next for RignSentinelAI

WebRTC Live Stream Ingestion: Expanding from snapshot polling to direct WebRTC video stream processing for real-time motion feed parsing.

Ring Alarm & Chime Integration: Triggering physical Ring Alarm base stations and Chimes to broadcast localized auditory warnings when critical threat vectors are verified on property bounds.

AgentCore Gateway Scaling: Deepening integration with AWS Bedrock AgentCore Gateways for MCP tool discovery and governance.

Geospatial Proximity Routing: Incorporating PostGIS or AWS Location Service to target camera feed queries strictly within geographically bounding alert polygons.

Built With

Share this project:

Updates

Submission history