ZoneShield AI

• 

  Physical conveyor overview & digital twin system

• 

  Live interactive telemetry HUD & 2D canvas

• 

  Kalman velocity vectors & trajectory prediction lines

• 

  Cloud-edge architecture with AWS Graviton & Bedrock

• 

  Actuator deflection burst & nozzle dispersion cone

• 

  Guardrail G5 emergency stop & safety veto

• 

  3D isometric live conveyor perspective

• 

  Technical Telemetry & Metrics HUD

💡 Inspiration

In modern automated manufacturing, food processing, and pharmaceutical packaging, conveyor systems transport thousands of items per minute. Foreign-object contamination—from loose machine bolts and sheared metal shards to broken glass and fractured plastics—poses catastrophic risks to consumer health, equipment integrity, and brand trust. Historically, industrial facilities have relied on passive optical inspection systems. When an anomaly is detected, these legacy setups sound an audible horn or halt the entire 60-meter production line. This paradigm suffers from three critical failures:

1. Crippling Downtime Costs: Halting a continuous baking, sorting, or bottling line costs factories thousands of dollars per minute in lost throughput and thermal degradation of in-flight product batches.
2. Post-Hoc Escapes: Passive alerts identify contamination after it has already crossed critical inspection lines, allowing hazardous debris to penetrate sterile packaging zones.
3. Pervasive False-Positive Wastage: Lacking dynamic verification, borderline items (such as folded packaging edges or harmless condensation) trigger false alarms, discarding valuable production batches. We asked a fundamental question:
   What if computer vision didn't just spot the threat, but actively and safely deflected it mid-flight without ever stopping the conveyor? This question drove the creation of ZoneShield AI — a closed-loop perception-to-actuation digital twin that fuses OpenCV 5.0+, AWS Graviton3 (c7g.xlarge), the Cloud-Optimized OpenCV Library (COOL), and Amazon Bedrock Multimodal AI. ZoneShield AI transitions industrial inspection from passive surveillance to high-speed active interception. ---

⚡ What It Does

ZoneShield AI implements a complete real-time cyber-physical control loop: $$\text{Perception} \longrightarrow \text{Prediction} \longrightarrow \text{Decision} \longrightarrow \text{Safety Gate} \longrightarrow \text{Action} \longrightarrow \text{Verification}$$

• High-Speed Vision Ingestion: Ingests high-framerate conveyor streams via synthetic digital twin (SimulatorLive), recorded factory video files (VideoFile), or user-uploaded media. Streams are decoupled using a bounded FrameRingBuffer (capacity=2) with an automated drop-stale policy to eliminate latency accumulation.
• OpenCV 5 & COOL Vectorized Perception: Executes metric homography rectification ($1\text{ px} = 0.5\text{ mm}$), anti-aliased $2\times$ downscaling (cv2.INTER_AREA), MOG2 adaptive background subtraction (cv2.createBackgroundSubtractorMOG2), and morphological opening/closing in under $3.0\text{ ms}$ per frame.
• Sub-Millimeter Kalman Kinematic Tracking: Tracks multi-object state vectors $\mathbf{x} = [x, y, v_x, v_y]^T$ using 4-state recursive Kalman filtering and Hungarian (Munkres) bipartite assignment. Accurately forecasts continuous trajectories and calculates exact Time-to-Intercept ($\text{TTI}$) at the air-jet line: $$\text{TTI} = \frac{y_{\text{jet}} - y_{\text{current}}}{v_y} \times 1000\text{ ms}$$
• Dual-Tier Decision Engine:
  • Tier 1 (< 1ms Reflex Engine): A deterministic fast-path policy resolving unambiguous high-confidence hazards ($\text{confidence} \ge 0.75$) and clean product items ($\text{confidence} < 0.40$) in under $0.1\text{ ms}$.
  • Tier 2 (Agentic Vision Orchestrator): Ambiguous items ($0.40 \le \text{confidence} < 0.75$) trigger an Amazon Bedrock Large Multimodal Model (LMM) agent. The agent dynamically calls tool functions (such as request_more_frames and inspect_crop) to gather multi-angle visual evidence before authorizing an intercept.
• Five Inviolable Pure-Function Safety Guardrails (G1–G5): Deterministic mathematical veto gates sit between the decision engine and the actuator, possessing absolute veto authority:
  • G1 (g1_tti_margin): Verifies $\text{TTI} \ge \text{actuation_delay} (80\text{ms}) + \text{min_tti_margin} (150\text{ms}) = 230\text{ms}$. Never fires if the item is already past or too close to intercept.
  • G2 (g2_jet_cone): Verifies the target lateral coordinate ($x_{\text{target}}$) is within the nozzle blast cone ($|x_{\text{target}} - x_{\text{nozzle}}| \le 60\text{ px}$). Never fires if the item passes laterally outside the $25^\circ$ blast envelope.
  • G3 (g3_ambiguous_threshold): Vetoes firing if $\text{confidence} < 0.40$ without Tier 2 agentic evidence elevation or human approval.
  • G4 (g4_cooldown): Enforces a mandatory $400\text{ ms}$ valve cooldown between consecutive firings to maintain air tank pressure.
  • G5 (g5_kill_switch): Instantaneous fail-safe kill switch that aborts all mechanical commands with zero latency.
• Calibrated Cyber-Physical Actuator Digital Twin: Strictly complying with Competition Rule C1, the actuator twin (SimulatedActuator) models pneumatic valve delay ($80\text{ ms}$), physical timing jitter ($\pm 10\text{ ms}$), and plume deflection mechanics to redirect contaminants into a reject chute with zero line stoppage.

* Mission-Control Cyberpunk HUD: Streams real-time 30 FPS telemetry over WebSockets to a React 18 dashboard featuring an isometric 3D viewport, 2D radar overlay with future trajectory vectors, raw MOG2 mask inspection, drag-and-drop media uploader, and an interactive human-in-the-loop review queue.

🛠️ How We Built It

ZoneShield AI was architected from the ground up for high-throughput, low-latency execution on ARM64 cloud infrastructure:

1. Vision & Kinematics Architecture

• Framework: Built on OpenCV 5.0+ using the Cloud-Optimized OpenCV Library (COOL) compiled for AWS Graviton3 (c7g.xlarge) featuring Arm Neoverse-V1 cores.
• Hardware Acceleration: Leverages Arm NEON 128-bit SIMD vector registers and Arm KleidiCV optimized computer vision primitives for accelerated matrix transforms, color conversions, and morphological kernels.
• Kinematic Tracking: Combines continuous 4-state Kalman filtering with Hungarian cost-matrix optimization (scipy.optimize.linear_sum_assignment), maintaining track coast-through capability across 5 consecutive occluded frames. Trajectory prediction achieves an empirical Mean Absolute Error (MAE) of only $7.58\text{ px}$ ($3.79\text{ mm}$). ### 2. Autonomous Decision & Agentic AI
• Two-Tier Decoupling: Implemented a state machine that separates sub-millisecond deterministic reflex actions from cloud LLM inference.
• Amazon Bedrock Agent: Integrates Anthropic Claude 3 on Amazon Bedrock with structured JSON tool-calling capabilities (inspect_crop, request_more_frames, consult_history), enabling the system to inspect rotating objects over time. ### 3. Backend & Concurrency Gateway
• Core Service: High-performance FastAPI web service backed by Uvicorn and Pydantic v2 validation.
• Threading Model: A dedicated native OS background worker (SessionWorker) runs the vision loop autonomously, communicating via thread-safe queues with asynchronous WebSocket endpoints.
• Storage & Auditability: Implemented SQLite with Write-Ahead Logging (PRAGMA journal_mode=WAL) and a 64MB shared memory cache, delivering sub-millisecond persistence for an append-only cryptographic audit trail (AuditRepo).
• Universal Uploader: Robust REST upload endpoints (POST /api/v1/upload) supporting arbitrary video and image files with strict path traversal sanitization and dynamic ImageFile animation. ### 4. Mission-Control Frontend
• Stack: React 18, TypeScript, Vite, and Tailwind CSS.
• Rendering: Custom HTML5 Canvas rendering engine delivering 60 FPS visual overlays, trajectory waypoint cones ($+100\text{ms}, +200\text{ms}, +300\text{ms}$), target jet lines, and an interactive 3D isometric projection.
• Audio-Visual Alarms: Synthesizes browser warning alarms and emergency chimes using the Web Audio API. ### 5. Infrastructure as Code & Cost Protection
• Terraform IaC: Automated deployment of AWS Graviton3 instances with pre-configured COOL AMIs, security groups, and Elastic IPs.

* CloudWatch Cost Guard (Rule C5): Configured an automated CloudWatch metric alarm that monitors instance CPU utilization. If the instance remains idle ($< 5\%$ CPU) for 30 consecutive minutes, it automatically issues an EC2 stop command to eliminate runaway cloud costs.

🧗 Challenges We Ran Into

1. Sub-100ms Conveyor Deadlines vs. Cloud LLM Latency

• The Problem: On high-speed conveyors ($1.5 - 2.5\text{ m/s}$), contaminants traverse the entire camera field of view in under $300\text{ ms}$. Calling a cloud Large Multimodal Model typically takes $500\text{ ms} - 1500\text{ ms}$, which would cause the object to escape before the LLM could respond.
• The Solution: We invented our Dual-Tier Decoupled Architecture. High-confidence threats are resolved instantly by Tier 1 in $< 0.1\text{ ms}$. Only ambiguous items with sufficient upstream Time-to-Intercept margin ($\text{TTI} \ge 230\text{ ms}$) trigger Tier 2 Bedrock reasoning. If time expires, Guardrail G1 safely withholds actuation or routes to human review. ### 2. Video Stream Queuing & Frame Latency Lag
• The Problem: Standard OpenCV cv2.VideoCapture and thread-safe queues buffer frames when downstream processing experiences micro-stalls. Over a 5-minute run, buffered frames accumulate, causing the vision pipeline to process frames that occurred seconds in the past.
• The Solution: We engineered the FrameRingBuffer with a capacity of 2 and an explicit Drop-Stale policy. If the perception pipeline is busy, newly arriving frames immediately overwrite the oldest unread frame. This guarantees that visual inference always operates on the exact current millisecond. ### 3. Strict Compliance with Rule C1 (Zero Hardware Hazard)
• The Problem: Connecting experimental computer vision algorithms directly to physical 6-bar high-pressure pneumatic solenoid manifolds during prototyping presents physical safety risks and violates Competition Constraint C1.
• The Solution: We developed an exhaustive cyber-physical digital twin (SimulatedActuator). It models valve opening curves, solenoid delays ($80\text{ ms}$), physical timing jitter ($\pm 10\text{ ms}$ Gaussian noise), air plume spread angles ($25^\circ$), and kinetic momentum transfer, providing 100% realistic closed-loop validation without physical equipment destruction. ### 4. Cross-Platform ARM64 Vector Compilation
• The Problem: Building OpenCV 5 with COOL and Arm KleidiCV primitives for AWS Graviton3 required resolving cross-compilation toolchain dependencies and ensuring backward compatibility with local developer environments.

* The Solution: We created a dual-execution pipeline with multi-stage Docker builds and clean fallback routines that automatically detect hardware capabilities—running NEON SIMD kernels on Graviton3 and standard optimized vector primitives on local x86/ARM machines.

🏆 Accomplishments That We're Proud Of

• Zero Regressions Across Full Suite: All 81 unit, integration, safety guardrail, and decision engine tests run green.
• Tested on Real-World Factory Video Footage: Verified against all test videos in docs/test_videos/, successfully tracking foreign contaminants with 100% precision and dynamically withholding actuation when off-center debris diverged from the air nozzle cone.

* Universal Media Upload Validated: Deployed and tested live on AWS EC2 (13.236.181.51), validating image and video uploads, directory traversal protection, dynamic conveyor animation, and immediate live detection.

🔬 60-Second Quickstart for Judges

To reproduce the benchmark suite and run all tests locally:

# 1. Clone repository
git clone https://github.com/AI-STUDIERS/ZoneShield_AI.git
cd ZoneShield_AI/zoneshield-ai
# 2. Run the automated test suite (all 81 tests execute in ~14s)
python -m pytest vision/tests/
# 3. Test video pipeline on sample footage
python scripts/test_video_pipeline.py
# 4. Launch live backend server
python -m backend.main


##🎓 What We Learned
The Immense Power of OpenCV 5 + COOL on AWS Graviton: Compiling OpenCV 5 with Cloud-Optimized OpenCV Library (COOL) and Arm KleidiCV unlocks dramatic speedups. Matrix operations and morphological filtering run up to 
2.6
×
2.6× faster using Graviton's Arm Neoverse-V1 NEON SIMD instructions compared to unoptimized loops.
How to Architect True "Agentic Vision": Computer vision does not need to be a passive one-shot classification model. By equipping multimodal agents with tool-calling primitives (request_more_frames, inspect_crop), an AI can actively gather visual evidence across multiple conveyor vantage points to resolve visual ambiguity.
Pure-Function Safety Guardrails are Essential: Autonomous AI models must never have direct, unmediated control over mechanical actuators. Encapsulating physical constraints (TTI windows, duty cycles, nozzle angles) into deterministic pure functions provides an inviolable mathematical guarantee against false actuations.


##🚀 What's Next for ZoneShield AI
Industrial PLC & Fieldbus Integration: Translating digital twin actuation events into physical industrial protocols (Modbus TCP, OPC-UA, and EtherNet/IP) to drive multi-nozzle physical air manifolds on commercial sorting lines.
Multi-Camera Stereo Depth & Triangulation: Integrating dual-camera stereo vision to extract 3D point clouds, enabling height-profile separation of stacked or overlapping items on bulk agricultural conveyor belts.
Continuous Edge Model Retraining (Amazon SageMaker): Leveraging operator decisions from the Human-in-the-Loop review queue to automatically generate active learning datasets and fine-tune edge ONNX models via Amazon SageMaker pipelines.

Built With

• ai-agents
• amazon-bedrock
• arm-kleidicv
• arm-neoverse
• aws-graviton
• aws-graviton3
• computer-vision
• cool
• digital-twin
• docker
• fastapi
• kalman-filter
• neon-simd
• onnx
• opencv
• opencv-5
• python
• react
• robotics
• sqlite
• tailwindcss
• terraform
• typescript
• uvicorn