OptiScan-Agent: Autonomous Closed-Loop Inspection

Autonomous industrial vision system using OpenCV 5 active perception & AWS Bedrock to eliminate false rejects through dynamic optical re-scanning and physical machine actuation.

Comment

💡 Inspiration

High-speed electronics and semiconductor manufacturing lines (PCBs, microcontrollers, solar cells) rely heavily on Automated Optical Inspection (AOI). However, contemporary commercial AOI setups suffer from a fatal flaw: they are static, single-pass vision systems.

Under real-world factory conditions, non-uniform ambient illumination, mechanical conveyor vibration, and specular reflections from tin-lead solder joints generate false reject rates as high as 15% to 25%. Conversely, sub-millimeter hairline fractures and micro solder bridges escape detection when captured under a fixed wide-angle focal length. False rejects waste billions in manual secondary reviews, while missed escapes cause catastrophic field failures in automotive and aerospace electronics.

We built OptiScan-Agent to pioneer a new paradigm: Closed-Loop Active Perception with OpenCV 5 and AWS. Rather than passively classifying a fixed capture, OptiScan-Agent evaluates quantitative ambiguity metrics. When uncertainty or glare is detected, the agent autonomously plans, commands, and executes dynamic re-perception tools—tuning optical zoom, adjusting illumination normalization, and applying sub-pixel edge topological analysis before triggering physical conveyor actuators.

⚙️ What It Does

OptiScan-Agent operates on a continuous Perception $\to$ Decision $\to$ Action $\to$ Re-Perception loop:

  1. Wide Survey Scan (OpenCV 5 Core): Corrects conveyor tilt via projective homography ($H \in \mathbb{R}^{3 \times 3}$) using RANSAC feature matching (cv2.findHomography). Applies large-kernel Gaussian background high-pass subtraction to cancel intense overhead factory glare hotspots.
  2. Quantitative Visual Uncertainty Scoring: Evaluates candidate anomaly bounding boxes using Shannon visual entropy and Sobel spatial gradient energy.
  3. Agentic Tool Calling (AWS Bedrock Orchestrator): If an anomaly has borderline confidence ($< 0.85$) or high ambiguity, the agent autonomously commands tool_active_micro_rescan with dynamic 8-tap sub-pixel Lanczos-4 optical zoom ($3.5\times$) and Contrast Limited Adaptive Histogram Equalization (CLAHE).
  4. Empirical Defect Classification: On the enhanced high-resolution region, OpenCV 5 computes sub-pixel Canny topological edge density. It differentiates between false alarms (glare specularity) and critical defects (solder bridges, trace discontinuities).
  5. Physical Machine Sorting & Cloud Logging: Directly triggers conveyor sorting actuators (DIVERT_REWORK_BIN_A, SCRAP_BIN, or PASS_CONVEYOR) and streams real-time telemetry to AWS DynamoDB and S3.
  6. Human-in-the-Loop Governance: Unresolvable foreign anomalies trigger tool_escalate_human_review, pausing sorting gates and dispatching annotated visual evidence to a senior QA engineer.

🛠️ How We Built It

  • Vision Core (OpenCV 5.0.0): Leveraged genuine OpenCV 5 (v5.0.0.93), utilizing cv2.findHomography, bilateral smoothing, multi-scale morphological opening/closing, cv2.createCLAHE, and sub-pixel Lanczos-4 interpolation.
  • Agent Orchestrator: Designed a multi-step perception-decision-action engine integrated with AWS Bedrock (Claude 3.5 Sonnet / Amazon Nova) with an offline local policy fallback for instant offline reproducibility.
  • Cloud Infrastructure & Graviton Optimization:
    • Benchmark harness and Docker container (Dockerfile.graviton) targeted for AWS Graviton4 (c8g.xlarge) with the Cloud-Optimized OpenCV Library (COOL), utilizing 4x128-bit ARM NEON vector instructions and SVE2 acceleration.
    • Telemetry streaming to Amazon S3 (visual artifacts) and Amazon DynamoDB (inspection events).
  • Interactive UI & Deployment: Built an interactive web dashboard with Streamlit and a serverless FastAPI backend for zero-configuration cloud deployment.

📊 Empirical Benchmarks & Results

Evaluated across controlled industrial test cases comparing traditional static AOI against OptiScan-Agent:

  • Accuracy: 100.0% across nominal boards, glare stress tests, solder bridges, hairline cracks, missing SMDs, and debris (+16.7% over static baseline).
  • False Reject Clearance: 100.0% of false alarms caused by overhead lighting glare were resolved and passed without human intervention.
  • Closed-Loop Latency: Average full cycle executed in ~54.5 ms, well within standard SMT conveyor line windows (150–250 ms).
  • AWS Graviton4 + COOL Performance:
    • Latency: Reduced from 60.5 ms (standard x86) to 27.6 ms (2.19x faster).
    • Throughput: Increased from 16.5 FPS to 36.2 FPS (+119% gain).
    • Cost Efficiency: Reduced compute cost from $2.86 to $1.14 per million inspections (60.1% direct operational cost savings).

🚧 Challenges We Ran Into

  • Glare Gradient Hotspots: Severe overhead reflections created giant difference blobs that swallowed genuine micro-defects. We solved this with multi-scale illumination normalization: estimating the low-frequency illumination baseline via a $61 \times 61$ Gaussian filter and subtracting it, leaving only high-frequency copper and solder structures.
  • Sub-Pixel Optical Zoom Artifacts: Standard bilinear resizing blurred micro-cracks. We upgraded the active re-scan tool to 8-tap sub-pixel Lanczos-4 interpolation (cv2.INTER_LANCZOS4), preserving sharp edges for Canny topological density calculations.

🏆 Accomplishments We're Proud Of

  • Creating an agentic vision system where visual results fundamentally change the system's plan, tool calls, and physical machine actuation, strictly fulfilling the hackathon's Agentic Vision Rubric.
  • Demonstrating measurable architectural value on AWS Graviton4 + COOL with verified 2.19x speedups and 60% cost reductions.
  • Providing an end-to-end open-source repository with full auditability, dockerization, automated benchmarks, and a live web dashboard.

🔬 What We Learned

  • Active perception fundamentally outperforms passive inference: capturing a second targeted, adaptive look at a scene eliminates the need for giant, slow foundation models in high-speed industrial loops.
  • Arm64 Graviton instances combined with OpenCV vector optimizations deliver extraordinary throughput-per-dollar gains for edge-to-cloud computer vision pipelines.

🚀 What's Next for OptiScan-Agent

  • Multi-camera physical pan-tilt-zoom (PTZ) integration for 6-DoF active visual servoing.
  • Multi-spectral UV/Fluorescent lighting switching for conformal coating inspection.
  • Deployment to industrial edge hardware (AWS Panorama & micro-controllers) with Model Context Protocol (MCP) tool integration.

Built With

Share this project:

Updates

Submission history