VisualOps AI

An AI safety copilot that helps operators investigate PPE risks with OpenCV 5 and Amazon Bedrock, checking extra video frames before creating an incident or requesting human approval.

Comment

Inspiration

Safety cameras capture activity, but turning footage into an operational decision still takes human attention. A person detection or a missing-helmet prediction leaves important questions unanswered: is the worker actually inside a restricted zone, is the visual evidence reliable, and does the situation need escalation?

I built VisualOps AI to explore a safety workflow where an agent can investigate those questions. The focus is deliberately narrow: restricted-zone entry and helmet compliance. The goal is to give a safety operator an evidence-backed case to review, with a visible record of how the system reached its decision.

What it does

An operator opens the dashboard and uploads a short video. OpenCV 5 detects people and PPE, tracks workers, and checks a configured restricted-zone polygon. Amazon Bedrock receives the structured observations and chooses investigation tools. It can request additional video frames, measure continuous time in the zone, and calculate risk before creating an incident.

Additional evidence changes the next action. Confirmed helmet presence can resolve a case. Ambiguous evidence requests human review. Confirmed missing PPE can produce an incident, with critical escalation left pending operator approval. Tool inputs, results, timestamps, evidence images, and risk factors are available in the interface.

The app includes a dashboard, inspection workspace, incident details, event timeline, and evaluation page. A separate Run demo option uses explicitly simulated visual observations with real Bedrock tool calls; uploaded videos use the actual OpenCV pipeline.

How we built it

I built the interface with Next.js, React, and TypeScript, and the API with Python and FastAPI. OpenCV 5 performs video decoding, preprocessing, ONNX inference, detection postprocessing, tracking, zone checks, and evidence annotation. Two pretrained YOLOv8 models handle construction people/PPE and specialist helmet/head detection. Model revisions, checksums, and preprocessing are recorded for reproducibility.

Amazon Bedrock Nova Lite uses the Converse API to choose structured tools. OpenCV tools perform the additional visual analysis and return new evidence. The tool executor enforces prerequisites: confirmed missing-PPE evidence and measured duration are required before risk assessment, and verified risk is required before incident creation.

The Docker container runs on Google Cloud Run with zero minimum instances to reduce idle hosting cost. Amazon S3 stores private videos and evidence, and DynamoDB persists runs and review decisions. Bedrock remains the investigation planner on AWS. Google service identity obtains temporary AWS credentials through STS, without static access keys. Artifact Registry stores release images deployed by digest, and Cloud Logging receives application logs. CloudFormation manages the retained AWS data resources. Media uses short-lived signed URLs; media lifecycle and run TTL are seven days.

Challenges we ran into

The main perception challenge was distinguishing an uncovered head from a helmet that the detector simply failed to see. I kept missing detections as unknown, added a specialist helmet model, and required agreement across additional observations. Crowding, occlusion, and person-to-head association remain limitations.

Another challenge was keeping the agent's decisions accountable. Instructions alone cannot enforce evidence requirements, so validation lives in the tool executor. An unsupported action is rejected, and service failures lead to a recorded human-review request. Non-root container startup also required writable nginx temporary paths; a container preflight now checks the runtime before deployment.

Evaluation required separating different kinds of evidence. Image detection, authored agent scenarios, and deployment smoke checks answer different questions, so the evaluation page labels each scope explicitly.

Accomplishments that we're proud of

I shipped an application using OpenCV 5 and AWS Bedrock with a complete upload-to-investigation-to-approval workflow and actual Bedrock tool execution. Remote checks verified OpenCV upload processing, DynamoDB persistence, encrypted S3 media retrieval, and conflicting review decisions. Sixteen Python tests cover evidence gates, ambiguity, tracking, duration, media paths, review concurrency, and AWS credential federation/refresh; the production frontend build also passed.

On 100 images from a published test split, head/helmet detection achieved 74.62% precision, 78.46% recall, and 76.49% F1. On 24 authored structured-event scenarios using actual Bedrock requests, decision accuracy was 79.17%, with an average decision latency of 3.007 seconds. Failure images and raw evaluation artifacts are retained.

These are scoped measurements, not a claim of field readiness. The specialist was trained on the same public dataset, split integrity has not been independently audited, and the replay scenarios do not establish real-video incident accuracy.

What we learned

A useful investigation needs explicit uncertainty and evidence requirements. Additional-frame tools are valuable when their results change the next action, and deterministic checks make the boundary between model planning and operational execution visible.

Reproducibility also requires more than a dependency list: model hashes, exact samples, tool traces, and failure examples make results inspectable. Human review remains part of the workflow when the evidence or service cannot support a reliable decision.

What's next for VisualOps AI

The next milestone is an external labeled-video benchmark covering normal activity, zone violations, missing PPE, and ambiguous footage. I plan to measure tracking, zone accuracy, real-video incident precision and recall, false alerts, and complete processing latency.

I also plan to improve tracking and head association in crowded or occluded scenes, add configurable zone settings, and build a repeatable demonstration from actual footage. Live camera integration can follow after those validation steps. The current release is a safety workflow MVP, not a certified safety system.

Built With

Share this project:

Updates

Submission history