• 

  system architecture flowchart

• 

  Visual preview with CCTV tracking and benchmark metrics

Inspiration

VisionGuard: Agentic Physical AI & Industrial Safety

Built for the OpenCV AI Competition 2026 Tracks: Agentic Vision & Best Use of COOL Live Interactive Demo: https://kbytehub0x.github.io/visionguard/ Open-Source Repository: https://github.com/kbytehub0x/visionguard

Inspiration Industrial hazard zones—such as overhead crane buffer bays, heavy machinery paths, and restricted chemical storage—suffer from high rates of false alarms caused by transient visual noise (shadows, dust, passing equipment reflections). Traditional computer vision systems either overwhelm safety operators with false positives or require cost-prohibitive, continuous 24/7 cloud GPU/LLM video streaming.

VisionGuard was built to solve this fundamental trade-off: How do we achieve deep, agentic reasoning and verifiable physical safety without ballooning cloud compute costs?

By coupling ultra-fast edge computer vision (OpenCV 5 accelerated via COOL on AWS Graviton3) with an autonomous agentic reasoning loop (Amazon Bedrock / Anthropic Claude 3.5 Sonnet), VisionGuard delivers verified hazard detection, closed-loop human oversight, and an empirical −58.4% cost reduction.

What It Does VisionGuard implements an autonomous four-stage perception, reasoning, and escalation pipeline:

1. Ultra-Fast Edge Triage (OpenCV 5 + COOL on AWS Graviton3):
2. Continuous CCTV streams are processed on ARM64 Graviton3 compute at an execution latency of 3.84 ms.
3. If a detected intrusion returns high, definitive confidence ($\ge 0.75$), it resolves immediately at the edge without triggering expensive cloud models.

4. If detection confidence is borderline or ambiguous (e.g., $0.68$), it triggers the agentic loop.

5. Agentic Reasoning (Amazon Bedrock / Claude 3.5 Sonnet):

6. Rather than outputting conversational text, Claude 3.5 Sonnet acts as an autonomous safety agent inside an AWS Step Functions state machine.

7. It analyzes the detection telemetry and dynamically calls the allowlisted tool: inspect_more_frames over a 4-second temporal window decimated at 5 FPS.

8. Secondary Physical AI Verification:

9. Kalman Filter: Tracks worker trajectory across the temporal window, confirming 3.8 seconds of stationary persistence in the hazard zone (eliminating transient false triggers).

10. YOLOv8 PPE Attribute Classifier: Evaluates the worker's safety equipment, confirming a missing hard hat with 0.94 confidence and escalating incident severity to HIGH.

11. Responsible AI: Human-in-the-Loop TaskToken Gate:

12. AWS Step Functions pauses execution using .waitForTaskToken, delivering annotated visual evidence frames to a human safety supervisor.

13. Only upon human supervisor approval is the task token resumed, automatically recording an auditable DynamoDB event and dispatching emergency Amazon SNS alerts.

Verified Benchmark Evidence (1280×720 @ 10,000 Frames) Evaluated against an on-demand x86 baseline using our committed, reproducible evaluation suite:

The Three Production Cost Optimization Levers:

1. Graviton3 Efficiency: ARM64 Neoverse-V1 cores paired with Arm KleidiCV vector extensions boost primary processing throughput by +61.5%.
2. AWS Fargate Spot: Secondary inspection tasks execute statelessly on Fargate Spot at a 70% discount ($0.009714/vCPU-hr).
3. Adaptive Frame Decimation: Amazon Bedrock dynamically downsamples secondary verification windows from 30 FPS to 5 FPS, evaluating 83.3% fewer frames while preserving Kalman temporal tracking.

How i Built It

• Computer Vision: OpenCV 5 compiled from source with Arm KleidiCV HAL vector acceleration (-D WITH_KLEIDICV=ON -D ENABLE_NEON=ON).
• Cloud Orchestration: AWS Step Functions state machine (VisionGuard-Agentic-Loop) managing execution transitions, retry backoffs, and task tokens.
• Agentic Reasoning: Amazon Bedrock runtime executing Anthropic Claude 3.5 Sonnet with ReAct tool-calling schemas.
• Tracking & Perception: Kalman Filter for multi-frame trajectory persistence and YOLOv8 for PPE compliance.
• Cloud Infrastructure: AWS ECS Fargate ARM64, Amazon S3, Amazon DynamoDB, Amazon SNS, and AWS CDK v2.

Challenges We Ran Into

• Compiling OpenCV 5 with KleidiCV on ARM64: Ensuring the Arm C Language Extensions (ACLE) and NEON SIMD vector dispatch compiled cleanly inside Docker required tuning multi-stage Dockerfile stages and pinned toolchains.
• Dynamic Compute Modulation: Balancing model confidence thresholds so that clear edge detections pass instantly in 3.84 ms, while ambiguous detections reliably trigger the Bedrock temporal verification loop without latency spikes.

Accomplishments That We're Proud Of

• Achieving an empirical, reproducible −58.4% net cost reduction backed by committed 10,000-frame JSON profiling data and automated one-command reproduction scripts (python3 benchmarks/compare_results.py).
• Constructing a true Physical AI closed-loop architecture where an LLM agent modulates physical compute tasks and camera verification rather than producing static text.
• Providing multiple ways for judges to experience the project: an interactive live web dashboard, a one-click Google Colab notebook, and a 1440p 60fps video walkthrough.

What's Next for VisionGuard

• Pan-Tilt-Zoom (PTZ) Camera Actuation: Giving the Bedrock agentic loop direct control to physically actuate PTZ cameras to zoom in on ambiguous zones.
• Multi-Camera Handover: Extending the Kalman tracking state across overlapping warehouse CCTV camera streams.

Built With

• amazon-bedrock
• aws-fargate
• aws-graviton3
• aws-step-functions
• claude-3-5-sonnet
• cool
• docker
• kleidicv
• opencv
• opencv5
• python