SentinelStore-AI

Sentinel AI is an always-on digital security chief that sees, hears, understands, and responds to retail incidents in real time — before humans even realize there’s a problem.

Comment

Project Story

Inspiration

Retail environments face a unique combination of challenges: high customer footfall, safety risks, theft, emergencies, and the need for fast, coordinated responses. Most existing systems are reactive, siloed, and heavily dependent on human operators monitoring multiple dashboards.

The inspiration behind the Retail Autonomous Incident Response System was to build an agentic AI system that can observe, reason, decide, and act autonomously in real time—similar to how a trained store manager would, but faster, more consistent, and always available. The goal was to reduce response time, minimize human error, and provide explainable, policy-backed decisions in critical retail incidents.

What We Learned

Building this project helped us gain deep, hands-on experience with:

  • Agentic AI systems using LangGraph and state machines
  • Multimodal AI reasoning, combining vision, audio, and video signals
  • Retrieval-Augmented Generation (RAG) for policy-aware decision-making
  • Human-in-the-loop (HITL) system design for safety-critical AI
  • End-to-end AI product engineering, from backend orchestration to frontend dashboards
  • Explainable AI, ensuring every decision can be justified using SOPs and policies

A key learning was that autonomous systems must balance intelligence with control. Full automation is powerful, but selective human intervention is essential for trust and safety.

How We Built the Project

The system is built as a multi-agent orchestration pipeline powered by LangGraph.

At a high level, the workflow follows:

  1. Incident Ingestion Incidents are submitted via a FastAPI backend using multimodal inputs:
  • Vision (images)
  • Audio (speech, noise)
  • Video (live or recorded feeds)

  1. Memory Retrieval with RAG A RAG engine retrieves relevant past incidents and store policies from a vector database. This provides historical and procedural context before making decisions.

  2. Multimodal Analysis Specialized agents process each signal independently:

  • Vision Agent for image understanding
  • Speech Agent for audio transcription
  • Video Agent for activity and anomaly detection

Their outputs are combined by a Fusion Agent to form a unified incident understanding.

  1. Risk Assessment and Decision Logic A Risk Assessment Agent computes: [ \text{severity} \in {1,2,3,4,5}, \quad \text{risk_score} \in [0,1] ] Based on thresholds and retrieved policies, the system decides whether:
  • It can proceed autonomously, or
  • Human-in-the-loop approval is required
  1. Planning and Execution A Planning Agent generates a step-by-step response plan using SOPs. The Response LLM converts this plan into executable actions such as:
  • In-store voice announcements
  • Emails via SendGrid
  • Phone calls via Twilio
  • Emergency escalation when required
  1. Monitoring, Reflection, and Learning After execution:
  • A Monitoring Agent tracks resolution status
  • A Self-Reflection Agent evaluates effectiveness
  • A Learning Agent updates long-term memory for future incidents

All decisions are accompanied by an Explainability Agent, ensuring transparency and auditability.

Challenges We Faced

  • Multimodal signal alignment Synchronizing and reasoning over vision, audio, and video data required careful state design to avoid conflicting interpretations.

  • Designing safe autonomy Determining when the system should act independently versus when it must pause for human review was a critical design challenge.

  • RAG relevance tuning Ensuring that retrieved policies and past incidents were contextually relevant required memory decay, severity boosting, and ranking strategies.

  • Scalability considerations The system needed to be designed with future production deployment in mind, including async execution, persistence, and monitoring.

  • Explainability vs performance trade-offs Providing detailed explanations without slowing down real-time responses required thoughtful agent separation.

Conclusion

The Retail Autonomous Incident Response System demonstrates how agentic AI, multimodal reasoning, and RAG can be combined to build real-world, safety-critical systems. The project goes beyond a chatbot or dashboard—it represents a shift toward AI systems that can reason, act, learn, and explain, while still keeping humans in control where it matters most.

This project serves as a strong foundation for real-world deployment in retail safety, operations, and incident management systems.

Built With

Share this project:

Updates