Gotcha!

Live detecting
Able to detect multiple people
Evidence tab that has history of suspicious suspects
Date and time stamps that correspond to incident
Sus!

Inspiration

Small businesses and independent vendors are frequently left vulnerable to theft because they lack the budget and space for traditional, industrial-scale CCTV systems. We built this project to bridge the gap between those expensive security networks and having no protection at all. By doing so, we aim to provide every business owner with an accessible, affordable tool to secure their livelihood.

What it does

Gotcha! is a real-time shoplifting detection system designed to help small businesses analyze live video feeds using machine learning to flag suspicious behavior and assign risk scores. To prevent false alarms, the system uses a sliding window heuristic that only triggers an alert when theft is detected across multiple consecutive frames. Once a threat is verified, Gotcha captures timestamped photo evidence and allows the shopkeeper to trigger an intelligent silent alarm that calls the assigned emergency contact with a descriptive explanation of the shoplifting that occurred via an automated phone call.

How we built it

Built as a lightweight, real-time system where a surveillance device streams live frames to a Flask backend via ngrok. The backend runs each frame through a shoplifting detection model, processes risk scores, stores evidence, and pushes real-time updates to a React dashboard using WebSockets, while also integrating Gemini, ElevenLabs, and Twilio for automated alerts.

Challenges we ran into

Risk Scoring: We had to develop a formula to calculate a dynamic 'risk factor' using our vision model's confidence scores.

API Coordination: The system involved at least 5 different services. The communication between those services led to complications, and it became hard to coordinate the API calls.

Pipeline Validation: Whether the audio on the phone call was actually generated by ElevenLabs, rather than Twilio's default fallback voice, was difficult to determine.

Accomplishments that we're proud of

Mitigating False Positives: We eliminated false positives by implementing a sliding window algorithm that requires multiple consecutive frame detections before triggering an alert.

Pipeline Synchronization: We successfully synchronized a complex, multi-step pipeline that seamlessly sequences evidence capture, Gemini AI analysis, and live dashboard broadcasting.

Asynchronous Call Integration: We resolved strict asynchronous timing constraints by orchestrating ngrok, ElevenLabs, and Twilio to reliably host and fetch the generated audio during live calls.

What we learned

Building Gotcha! served as a technical crash course, teaching us how to implement ngrok for network tunneling, WebSockets for real-time data streaming, and complex third-party APIs like Gemini and Twilio. Beyond the technical skills, we also learned how to successfully architect and divide a full-stack project across a four-person team under intense time constraints.

What's next for Gotcha!

Machine Learning Optimization: We plan to upgrade our base model and implement reinforcement learning that automatically retrains the system using flagged real-world detections.

Inventory Integration: We aim to cross-reference stolen items with store databases to provide owners with real-time loss valuations and track historical theft totals.

Conversational Dispatch: We intend to upgrade our emergency response system from a one-way alert to a live, two-way conversational AI agent using Twilio's advanced platform.

Hardware Scalability: We will scale the system to support multiple simultaneous camera feeds and diverse hardware types to accommodate complex retail environments.