SentinelVision

• 

  aws_s3_uploaded_keyframe

• 

  aws_cloudwatch_metrics_console

• 

  aws_dynamodb_items_console

• 

  aws_s3_trace_console

• 

  aws_s3_snapshot_console

• 

  evaluation_edge_case_report

• 

  incident_workflow_live

Inspiration

Industrial environments bring people, vehicles, forklifts, and machinery into shared spaces where hazards can develop quickly. We were inspired by the possibility of using computer vision not just to detect what is happening in a scene, but to help identify developing risks and give safety teams clearer, more timely information.

We wanted to explore how AI could move beyond simply drawing boxes around objects and help turn video observations into meaningful safety decisions.

What it does

SentinelVision is an AI-powered industrial safety monitoring prototype that analyzes video to identify objects, track their movement, estimate potential collision risks, and manage safety incidents.

It combines object detection, object tracking, time-to-collision (TTC) estimation, and a risk-assessment layer that assigns risk levels from LOW to CRITICAL. Its incident-management workflow consolidates repeated alerts, manages incident states, and lets a human operator acknowledge, escalate, or dismiss an incident.

The project also includes AWS integration components for cloud storage, incident-state persistence, and monitoring. The live video-processing pipeline currently runs locally, while the AWS components support the cloud integration work.

How we built it

We built SentinelVision using Python, OpenCV 5, a YOLOv8-based object detector, and an IoU-based tracking component. OpenCV handles image and video processing, while the detection and tracking pipeline supplies observations for risk assessment.

We then developed a time-to-collision estimation component and an incident-management layer to translate detections and risk assessments into tracked incidents. A FastAPI application exposes the system's web interface and endpoints, including controls for human incident review.

For cloud integration, we implemented components using Amazon S3, Amazon DynamoDB, and Amazon CloudWatch. We also developed evaluation scripts and supporting evidence to test detection behavior, alert consolidation, incident transitions, and selected cloud operations.

Challenges we ran into

One major challenge was distinguishing meaningful hazards from noisy or short-lived detections. Object detection alone is not enough: the system must follow objects across frames and manage alerts over time without overwhelming an operator.

We also had to work within CPU-only hardware constraints, which limit the speed of the complete video-processing pipeline. Another challenge is that collision estimates derived from image coordinates are affected by camera perspective and are not equivalent to calibrated physical measurements.

Finally, integrating cloud services while keeping the current live video pipeline local required us to separate what was tested independently from what is already connected end to end.

Accomplishments that we're proud of

We are proud to have developed a working pipeline that connects video analysis to risk assessment and incident management, rather than stopping at object detection.

We implemented incident-state handling, alert consolidation, and human override controls, giving operators a way to review and manage generated incidents. We also built AWS integration components and created repeatable evaluation artifacts to make our claims more traceable.

In one documented video evaluation, SentinelVision detected both of the two annotated encounters and consolidated four fragmented alerts into two incidents. This is an encouraging result on that particular clip, not a guarantee of performance across all environments.

We are also proud of taking a limitations-aware approach: we distinguish isolated component performance from end-to-end throughput and treat SentinelVision as an advisory prototype rather than a certified safety system.

What we learned

We learned that useful computer vision involves much more than detecting objects. Reliable monitoring also requires tracking, temporal reasoning, alert management, failure testing, and a clear process for human review.

We learned to evaluate the entire pipeline separately from individual components and to be careful about how metrics are interpreted. For example, a geometric TTC estimate from image coordinates is not a calibrated prediction of physical time to collision.

We also learned the importance of reproducible tests, evidence-backed reporting, and clearly communicating the difference between implemented functionality, tested integrations, and future work. AI-assisted tools supported parts of our development and documentation workflow, including the use of Google Gemini NotebookLM to help prepare our project presentation video.

What's next for SentinelVision

Our next steps are to evaluate SentinelVision across more video samples, environments, camera angles, lighting conditions, and occlusion scenarios. We want to improve tracking continuity, investigate better motion estimation and camera calibration, and measure performance under realistic operating conditions.

We also plan to strengthen end-to-end AWS deployment and monitoring, improve the reproducibility of our evaluation process, and explore how safety teams could integrate incident review into existing workflows.

SentinelVision remains an advisory research prototype. Before real-world safety deployment, it needs broader validation, calibrated measurements where required, and appropriate safety and operational review. Our long-term goal is to help safety teams identify potential hazards earlier and make better-informed decisions while keeping people in control.

Built With

• agenticvision
• amazon-web-services
• amazoncloudwatch
• amazondynamodb
• amazons3
• artificialintelligence
• computervision
• fastapi
• humaninaloop
• industrialsafety
• objecttracking
• opencv
• python
• videoanalytics
• yolov8