Self-Correcting Vision Agent for Industrial Safety Cameras

A safety-camera agent that checks if each frame can be trusted, fixes dark or blurry frames with OpenCV 5 tools (local or AWS Lambda), and hands unclear cases to a person instead of guessing.

Comment

Inspiration

In steel plants and refineries, safety cameras rarely get a clean picture. Night shifts, floodlight glare, dust, steam and wind-shaken poles all make workers hard to see. A normal detector still gives an answer on every frame, so a missed helmet in a dark image looks exactly like an empty walkway, and a frozen camera looks like a quiet shift.

We wanted a system that asks a different question first: can this frame be trusted?

What it does

  • Checks the camera itself for a frozen feed, signal loss, a dirtying lens or lighting drift.
  • Measures every frame with OpenCV 5 (brightness, sharpness, noise, glare) and detects workers and helmets.
  • Gives a verdict with a reason: Normal, Abnormal or Insufficient, with the measured value behind it. For example: "too dark: brightness 37 (needs at least 60)".
  • Repairs only what it can: a versioned policy maps each reason to a tool: CLAHE for low light, the sharpest earlier frame for blur, denoising for noise, region upscaling for distant workers, or a vote across earlier frames.
  • Verifies every repair: it measures again and keeps the repair only if the problem actually got better (brightness up for a dark frame, noise down for a grainy one).
  • Asks a person when it should: after two rounds, or when no tool can help (glare, a blocked view, a worker cut by the frame edge), the case goes to a human review queue with the frame and the evidence.
  • Confirms before alarming: violations must persist for each tracked worker, one alert is raised per incident, and a hidden worker or dead camera is never reported as all clear.
  • Records every decision as a replayable trace.

How we built it

OpenCV 5 does the vision work: cv2.dnn runs YOLOX for people and YOLO-World (exported to ONNX) for helmet vs. bare head on each worker crop, with zoomed tiles for distant workers. CLAHE, non-local-means denoising, Laplacian sharpness and frame differencing handle repair and camera health.

The core is a reflection loop:

verdict engine → policy table → tool → improvement guard → decide or escalate

The policy picks a tool for the measured reason, and the improvement guard re-measures the frame instead of assuming the repair worked.

The live runtime adds zone rules, dwell limits, per-worker tracking, confirm-before-alarming, and drift checks against each camera's commissioned picture.

On AWS, the low-light repair tool runs as an AWS Lambda container image with OpenCV 5, exchanging frames through Amazon S3. Switching a tool between local and Lambda is one line of configuration. The live demo runs on Amazon EC2 behind an Application Load Balancer.

A web dashboard shows the live feed, the exact frame the agent judged, each repair round with before/after values, alerts, and the human review queue.

The project has over 2,000 automated tests, with linting and type checks on every change.

Challenges we faced

Helmet detection without training. The zero-shot helmet model finds a worn helmet in only about half of the frames. Instead of hiding this, we reduced the false alarms it causes with a re-check on earlier frames and a confirmation timer.

Real time on a CPU. The agent judges the newest frame whenever it is free, so the judged frame can lag behind the live picture. We chose to show that lag instead of hiding it.

Tracking at about one look per second. A walking worker can move more than their own width between looks, which broke tracks. We added distance-based track continuation.

What we learned

The biggest lesson: "I can't tell" is a valid and useful answer. A vision system should not be forced into a confident prediction from a frame that is too dark, blurry or noisy. Measuring picture quality before trusting a detector prevents silent failures.

Repairs must be verified, not assumed. For example, brightening a blurred frame can raise its sharpness score without making it any sharper, so the system measures the result.

And running a tool locally or in the cloud should be a configuration choice, not a rewrite.

What's next

  • Fine-tune the helmet detector on site-specific data.
  • Move more repair tools to Lambda.
  • Validate the DynamoDB trace store on live AWS.
  • Add HTTPS with a custom domain.

Built With

Share this project:

Updates

Submission history