Inspiration
Workplaces such as warehouses, logistics facilities, and manufacturing plants use cameras to monitor safety, but identifying what actually happened during a potential incident can still require time-consuming manual investigation. A motion alert or a single object detection does not prove that a person crossed a restricted boundary.
We built ZoneGuard around a different idea: computer vision should not stop at detecting a possible event. It should investigate the visual evidence, determine what additional information is needed, and distinguish a supported incident from an uncertain situation.
Our goal is to make workplace video investigation more transparent, efficient, and evidence-driven.
What it does
ZoneGuard is an Agentic Visual Safety Inspector that analyzes recorded workplace video to investigate potential restricted-zone incidents.
The system combines OpenCV 5, a pretrained YOLOX ONNX detector, object tracking, configurable polygon zones, and MCP-based tool orchestration.
Its workflow is designed to:
1- Analyze video frames and identify candidate objects. 2- Track their movement relative to configured restricted zones. 3- Investigate possible entry or exit events using temporal and spatial evidence. 4- Select follow-up analysis steps based on previous visual results. 5- Produce an outcome of ALERT, CLEAR, or REVIEW. 6- Generate annotated video and a JSON investigation report for inspection. And More. The core principle is that a detection is not automatically a violation. When a crossing cannot be established confidently, the system should preserve the uncertainty rather than manufacture an incident.
How we built it
ZoneGuard uses Python and OpenCV 5.0.0 for the vision pipeline. The detector uses the YOLOX INT8 ONNX model through OpenCV DNN, with model-integrity verification through SHA-256.
The investigation logic incorporates multi-zone polygon geometry, footpoint-based occupancy, tracking history, temporal gaps, and transition verification.
An MCP server exposes registered investigation tools, while the controller uses visual analysis results to guide follow-up operations. The project includes annotated MP4 generation, JSON reporting, and a web-based investigation interface.
The implementation is being developed as a local-first application so the complete vision and investigation workflow can be tested before cloud deployment.
What makes it different
ZoneGuard focuses on evidence-driven investigation rather than detection alone.
Its intended differentiators are adaptive visual inspection, transparent tool-call traces, explicit uncertainty handling, and inspectable incident evidence.
Instead of treating every detection as a confirmed safety event, the investigation controller can seek additional evidence or leave an ambiguous event for human review. This creates a more auditable approach to safety monitoring and gives operators a way to understand how an outcome was reached.
Challenges we ran into
One of the hardest problems is distinguishing a person who appears inside a restricted zone from a person who has actually crossed its boundary. Detection gaps, occlusion, tracking changes, boundary jitter, and insufficient video evidence can all produce misleading conclusions.
Another challenge is making the agent genuinely evidence-driven. The visual output must influence the next tool call and subsequent decision; a fixed pipeline followed by an AI-generated explanation is not sufficient.
We addressed these challenges by developing zone-transition verification, tracking-related logic, registered MCP tools, explicit review outcomes, and controller branching tests. The next major challenge is validating these components against a manually labelled video dataset.
Accomplishments we're proud of
The current development notes report a working OpenCV 5.0.0 runtime, YOLOX inference through OpenCV DNN, model hash verification, MCP integration, and multi-zone analysis.
The prototype has processed a supplied video end to end and generated an annotated MP4. The reported test run passed five automated tests. The sample investigation returned REVIEW because the available evidence did not establish a verified zone crossing.
We consider this a useful foundation for a system that prioritizes evidence and transparent decisions over unsupported alerts.
What we learned
Computer vision detection and event verification are different problems. A model can identify an object without proving that a meaningful safety event occurred.
We also learned that agentic behavior must be demonstrated through real execution: previous visual results must change a subsequent tool call, and the actual tool result must influence the investigation outcome.
Finally, a successful demonstration is only a starting point. Reliable safety applications require labelled evaluation data, negative and ambiguous test cases, reproducible results, appropriate safeguards, and clear documentation of limitations.
What's next for ZoneGuard
Our next priorities are to create a manually labelled evaluation dataset, measure event-level precision and recall where supported by the data, test difficult tracking and boundary cases, and strengthen the end-to-end investigation experience.
We also plan to complete the reviewer workflow, improve evidence traceability, validate the application in a clean environment, and deploy a meaningful vision-processing component on AWS.
Our long-term vision is to help workplace safety teams investigate video events faster while retaining human control over uncertain or consequential decisions.
Built With
- agentic
- ai
- amazon-web-services
- api
- css
- dnn
- docker
- fastapi
- ffmpeg
- github
- html
- javascript
- json
- mcp
- numpy
- object-tracking
- onnx
- opencv-5
- pytest
- python
- rest
- yolox
Log in or sign up for Devpost to join the conversation.