Inspiration
Warehouse safety inspections are still surprisingly manual. A safety officer walks the floor, checks whether workers wear PPE, looks for blocked aisles, and logs incidents after the fact. We wanted to build something that closes the loop: a system that sees a hazard, decides what to do next, and acts—all with a human in the loop when it matters.
The competition’s emphasis on Physical AI and Generative AI systems where visual understanding drives useful decisions matched exactly what we had been prototyping. We were also curious whether we could make the vision workload cheap and fast enough for real deployment by using COOL (Cloud-Optimized OpenCV Library) on AWS Graviton. That combination—agentic vision plus cloud-optimized OpenCV 5—became VisionLoop.
What it does
VisionLoop is an agentic visual inspection system for warehouse safety. It ingests live or recorded camera feeds, uses OpenCV 5 for substantive image and video analysis, and runs a multi-step perception → decision → action loop on AWS.
At a high level, VisionLoop:
Perceives frames with OpenCV 5: person detection, PPE compliance (helmet/vest), aisle obstruction, and zone intrusion.
Scores each event with confidence and severity, Decides via an agent: if severity exceeds a threshold τ τ, the agent chooses an action—send an alert, request human approval, re-check a different camera angle, or trigger a PTZ preset.
Acts and records the outcome. Crucially, the visual evidence changes what the agent does next. A chatbot that only explains a fixed detection would not qualify; VisionLoop’s OpenCV 5 output directly alters the subsequent tool call or approval request.
The system runs a meaningful component on AWS: the vision workload executes on AWS Graviton using COOL, while orchestration, storage, and observability use AWS services. We pursued both the Agentic Vision path and the Best Use of COOL path.
How we built it
Architecture We kept the architecture deliberately reproducible:
Ingest: RTSP or uploaded video → S3.
Vision worker: containerized OpenCV 5 + COOL on ECS with Graviton tasks. COOL accelerates core operations such as resizing, color conversion, and DNN preprocessing on the Arm path.
Agent orchestrator: a Python service on Lambda/ECS that receives structured perception results and decides the next action. It can call tools: send_alert, request_human_approval, recheck_camera, log_incident.
Human-in-the-loop: Step Functions waits for approval when the agent requests it.
Observability: CloudWatch metrics, structured logs, and a simple web dashboard.
We used OpenCV 5’s DNN module for detection and tracking, and COOL for the hot path. The agent loop is explicit: perception outputs are converted to a JSON “evidence packet” that includes bounding boxes, confidence scores, and frame timestamps. The agent uses that packet to select the next tool call.
Reproducibility and evaluation We pinned dependencies, containerized the vision worker, and documented build/deploy steps. For COOL, we compared the same workload on x86 and Graviton. We measured throughput, p95 latency, and cost per 1,000 frames. We also evaluated the agent loop on a small labeled set of warehouse clips: task success, false alerts, and human-approval rate.
Challenges we ran into
Keeping the agent genuinely agentic. Early versions were just “detect → alert.” We had to redesign the loop so that OpenCV 5 results changed the next decision—e.g., low confidence triggers a re-check from another camera, and a PPE violation in a high-traffic zone requires human approval. That forced us to define clear tool schemas and failure handling.
COOL integration and benchmarking. Getting COOL to execute the claimed core workload on the Arm path took careful container configuration. We also had to make the baseline fair: same instance family, same input resolution, same model.
Latency vs. accuracy. Real-time inspection pushed us to downsample and batch frames. We used a confidence threshold
τ
0.55 τ=0.55 after sweeping the precision–recall curve. Responsible operation. Safety alerts can have real consequences. We added human approval for high-severity actions, audit logs, and explicit limitations in the report.
Accomplishments that we're proud of
A working end-to-end system: video in → OpenCV 5 perception → agent decision → action → audit trail.
A reproducible COOL benchmark on AWS Graviton showing measurable speed and cost improvements over our x86 baseline.
A clear agent trace demonstrating that OpenCV 5 output changed a later tool call (e.g., re-check camera → request approval → alert).
A judge-accessible web endpoint and a 5-minute demo video showing the loop in action.
Documentation that another team could redeploy, including pinned dependencies, architecture diagram, and failure cases.
What we learned
Agentic vision is about the loop, not the chatbot. The hard part is making visual evidence actionable and observable. We learned to design tool calls around uncertainty: when confidence is low, the agent should gather more evidence rather than guess.
COOL on Graviton is practical for vision workloads. The Arm path is viable for production when the hot operations are optimized. Benchmarking discipline matters more than raw numbers.
Responsible-use design is a feature. Human approval, audit logs, and clear limitations made the system more trustworthy and easier to evaluate.
Reproducibility is a competitive advantage. Pinned dependencies, containerized workers, and a clear baseline made our results defensible.
What's next for VisionLoop
Multi-camera active perception: let the agent choose which camera to inspect next based on uncertainty and coverage.
Broader domains: extend the same loop to healthcare safety, agriculture monitoring, and accessibility assistance.
Better evaluation: larger labeled datasets, failure-case analysis, and task-success metrics across agent policies.
Production hardening: autoscaling, cost dashboards, and stronger human-in-the-loop UX.
Deeper COOL optimization: profile more OpenCV 5 operations on Graviton and document hybrid x86/Arm deployment patterns.
VisionLoop showed us that a well-designed perception–decision–action loop can turn OpenCV 5 from a vision library into a real-world safety system—and that cloud-optimized OpenCV on AWS Graviton makes it practical to run.
Log in or sign up for Devpost to join the conversation.