Sentinel AI — Factory Safety Copilot

Sentinel AI turns automotive factory video into time-coded, evidence-backed safety incidents and actionable improvement reports using local vision and optional GPT-5.6 verification.

Comment

Inspiration Automotive factories already have extensive camera coverage, yet safety teams cannot continuously monitor every production area. Traditional computer-vision demonstrations often stop at drawing a bounding box. We wanted to go further: convert visible evidence into a time-coded, explainable safety incident that helps a qualified safety professional decide what to do next. Our goal was to connect video analytics, site-specific safety policies, temporal evidence, incident management, and improvement reporting in one practical workflow. What it does Sentinel AI analyzes uploaded factory videos and active browser-camera feeds for visually observable occupational safety risks. It can: Detect and anonymously track workers and vehicles. Apply user-drawn safety zones to both uploaded video and live camera analysis. Identify candidate events such as missing helmets, restricted-zone entry, and worker–vehicle proximity. Display risks directly on video using red overlays such as NO HELMET and ZONE BREACH. Require multi-frame and minimum-duration evidence to reduce false alarms. Mark visually ambiguous situations as inconclusive instead of forcing a decision. Optionally send chronological evidence frames and the applicable site policy to GPT-5.6 for structured verification. Produce annotated MP4 videos, evidence images, JSON results, and management-ready PDF reports. Support acknowledgement, assignment, resolution, and false-alarm feedback for incidents. A deterministic guided demonstration and precomputed evidence package allow judges to evaluate the complete workflow without API credentials or credits. How we built it We built Sentinel AI as a local-first FastAPI application with a responsive HTML, CSS, and JavaScript interface. The computer-vision pipeline uses OpenCV, Ultralytics models, pose estimation, object detection, polygon zones, and anonymous temporal tracking. Uploaded videos are decoded sequentially, while live-camera analysis processes the newest available frame to avoid an expanding queue. Candidate events pass through policy and temporal validation rules. For example, a missing-helmet event requires multiple supporting frames, sufficient duration, visible head geometry, and a valid torso pose. The same event concepts and configured zones are shared between uploaded video and active-camera workflows. GPT-5.6 is an optional reasoning layer. When explicitly enabled, Sentinel sends a bounded set of chronological evidence frames together with the selected site policy through the OpenAI Responses API. The response is validated against a strict Pydantic schema containing event type, severity, confidence, visible and missing PPE, an evidence-based explanation, and a recommended immediate action. SQLite stores incidents and job state. A persistent single-worker queue prevents duplicate concurrent analyses, while stop-and-reset controls let the operator safely cancel a job. Challenges we ran into The largest challenge was reducing false alarms. A single detection frame is not enough to establish a safety-policy violation, especially when a helmet may be hidden by occlusion, camera angle, or low image quality. We introduced temporal confirmation, visibility checks, confidence thresholds, event deduplication, and explicit inconclusive states. Tracking was another challenge. Scene changes and short detection gaps can fragment one physical worker into multiple anonymous track IDs. We therefore report zone-level policy events while retaining affected track IDs only as supporting evidence. Factory-specific objects also exposed the limitations of general-purpose detection models. Classes such as cars and trucks are broadly supported, but specialized PPE, forklifts, automotive components, and tools require compatible models and site-specific validation. Finally, we needed a reliable competition demonstration that did not depend on internet connectivity, API credits, camera availability, or a live factory. We solved this with a controlled automotive safety video, gold time-code annotations, downloadable evidence, and a credential-free guided workflow. Accomplishments that we're proud of We are proud that Sentinel AI became a complete safety workflow rather than a standalone model experiment. Our key accomplishments include: A unified policy engine for uploaded video and active-camera analysis. Red, time-coded safety overlays on actual processed video. Multi-frame PPE, zone-breach, and vehicle-proximity event logic. Optional policy-aware GPT-5.6 verification with structured outputs. Anonymous tracking without facial recognition. Downloadable annotated video, evidence images, JSON, and PDF reports. Incident acknowledgement, assignment, resolution, and false-alarm handling. A single-job queue with duplicate-click protection and stop/reset controls. A controlled gold dataset and reproducible event-level evaluation. 34 passing automated tests. A complete judge demonstration that works without API credentials. In our controlled helmet-policy scenario, Sentinel matched the annotated 20–40 second violation with one true positive, zero false positives, zero false negatives, and a temporal IoU of 0.998. We treat this strictly as pipeline validation on a small controlled scenario—not as a claim of production-wide accuracy. What we learned We learned that occupational-safety AI requires more than object detection. A useful system must represent uncertainty, connect observations to site policy, preserve evidence, minimize alert fatigue, and keep a human responsible for the final decision. We also learned that local vision and language-model reasoning work best as separate layers. Local processing provides fast, privacy-conscious candidate detection, while GPT-5.6 can be reserved for ambiguous or high-value events that benefit from policy-aware reasoning. This architecture improves reliability and keeps API usage intentional and cost-controlled. Most importantly, we learned to distinguish what a camera can prove from what it cannot. Noise exposure, electrical isolation, chemical concentration, equipment condition, and procedural compliance may require sensors, records, or physical inspection. Sentinel reports only what is visibly supported and clearly communicates its limitations. What's next for Sentinel AI — Factory Safety Copilot Our next step is a monitored automotive-factory pilot using consented, site-specific footage. We plan to: Expand the independently reviewed gold dataset across different factories, camera angles, lighting conditions, PPE types, and occlusion levels. Train or integrate factory-specific models for helmets, safety vests, forklifts, automotive parts, tools, and hazardous obstructions. Calibrate confidence and duration thresholds separately for each camera and safety zone. Add cross-camera event correlation without introducing identity recognition. Integrate approved sensor and operational data for risks that video alone cannot verify. Measure precision, recall, latency, false-alert rate, and operator acceptance in realistic conditions. Add multilingual safety reports and configurable site policies. Introduce role-based deployment controls, audit exports, and integrations with existing safety-management systems. Sentinel AI will remain decision support rather than autonomous enforcement. Our objective is to help qualified safety teams review evidence faster, prioritize genuine risks, and take safer action.

Built With

Share this project:

Updates