InspectLoop

An inspection agent for PCBs: OpenCV 5 checks the capture, compares it with a golden board, scores a DNN patch memory, zooms where the heatmap points and asks a person when unsure. Runs on AWS.

Comment

Inspiration

At Rui Codex we build software for small manufacturers and distributors. On an electronics line every board gets a verdict in about a second. A missed defect ships to a customer, and a false reject scraps or reworks a good board. Unsupervised anomaly detectors fit a new product well, because they learn from good boards only, but they give one score and one threshold. Every board near that threshold is a coin flip, and a blurred or badly exposed capture looks like a defect. We wanted an inspector that knows when it is unsure, looks again where it matters, and hands the hard cases to a person.

What it does

InspectLoop is an inspection agent. It looks at a board, decides what to look at next and asks a person when it is not sure. Each step is an OpenCV 5 tool, and the visual result of each step chooses the next tool call:

  1. assess_capture checks sharpness (variance of the Laplacian, Tenengrad), a directional gradient-energy ratio for motion blur, brightness and clipping against limits learned from good captures. A bad frame is never scored: the agent calls request_recapture with a hint (refocus or hold still, exposure), at most twice.
  2. classical_check registers the capture to a per-type golden board (ORB, then SIFT, then ECC when registration fails), equalises it with CLAHE in Lab and differences it. A difference far outside the good boards is a FAIL without running the DNN (27 of 602 held-out boards).
  3. detect_anomalies runs a WideResNet-50 trunk in the OpenCV 5 DNN module against a patch memory of good boards, and fuses its score with the classical one on a common scale.
  4. In the uncertain band, zoom_inspect takes a 2x full-resolution crop where the anomaly map peaks.
  5. If the evidence is still mixed, escalate_to_human stops with both heatmaps and the trace.

A FAIL is only a proposal: it creates a non-conformance record that a person approves or overrides before any scrap or rework. Every step (tool, arguments, rationale, observation, latency) is shown in the UI, written to CloudWatch and stored in DynamoDB.

How we built it

  • OpenCV 5.0.0 (opencv-python==5.0.0.93, pinned) for all image analysis: Laplacian, Sobel, cvtColor, createCLAHE, ORB_create, SIFT_create, findHomography(USAC_MAGSAC), findTransformECC, warpPerspective, absdiff, GaussianBlur, morphologyEx, connectedComponentsWithStats, and dnn.readNetFromONNX for the backbone. Torch is used once, offline, to export the ImageNet backbone to ONNX. OpenCV 5 DNN reproduces the torch features (maximum relative error 1.5e-6, cosine 1.000000).
  • Calibration without defect labels. Every threshold comes from 100 held-out good training boards per type: accept at the 90th percentile, decide at the 99th, and a sure-fail level above that.
  • The agent policy (evidence-v2) is deterministic, so every reported number can be reproduced exactly. The same interface accepts Claude on Amazon Bedrock as the planner (Converse tool use), behind guardrails that validate every proposal against the tool whitelist and refuse a PASS that rests on the cheap classical evidence alone.
  • On AWS (us-east-1, one CDK stack): API Gateway (HTTP API) and a Lambda proxy that adds an origin secret from Secrets Manager and wakes the worker; an EC2 c8g.xlarge (Graviton4) worker on Ubuntu 24.04 arm64 with stock OpenCV 5; S3 for the code and model release; DynamoDB for records and human decisions; CloudWatch for per-step traces; an EventBridge-driven Lambda that stops the worker after 30 idle minutes. A direct call to the worker without the secret gets 403. Budgets alert at $20 and $40. A fresh-instance deploy was verified: the stack replaced the worker, which bootstrapped from S3 unattended and served on the first poll.
  • The UI is a single page that works on desktop and phone (the photo input opens the rear camera), follows the system light or dark mode, announces results through an aria-live region and labels every verdict colour with text.

Evaluation

Data: Amazon VisA (CC BY 4.0), pcb1 to pcb4, official 1cls split, 802 test boards. The design was fixed on pcb1 only, the code was frozen (commit ffdb53e), and pcb2 to pcb4 were evaluated once: 602 held-out boards, 300 of them anomalous.

arm (held-out pcb2-4) precision recall F1 false-reject rate
classical alone 0.942 0.487 0.642 0.030
single DNN pass 0.951 0.587 0.726 0.030
fused, both always 0.961 0.657 0.780 0.026
agent, escalated boards held 0.953 0.747 0.837 0.036
agent, auto-decided only (77.1 % coverage) 0.969 0.553 0.704 0.010
  • Mean image AUROC over the held-out types: single DNN pass 0.943, classical 0.926, fused 0.961 (per board: pcb2 0.962, pcb3 0.930, pcb4 0.989; pcb1, the development board, 0.963).
  • The agent decides 77 % of boards without a person, at 96.9 % precision and a 1.0 % false-reject rate.
  • On simulated degraded captures, a single DNN pass flagged every degraded good pcb4 board as defective, while the quality gate caught 95.9 % of the degraded frames across the held-out boards.
  • LLM planner on Bedrock (60 held-out boards, half anomalous): Claude Opus 4.6 agreed with the fixed policy on 86.7 % of verdicts, with F1 0.881 against 0.900 (escalations held), 30.5 s against 0.87 s median per board and about $0.063 per board. The guardrail rejected 30 proposals, all of them a PASS from the classical evidence alone, and forced the DNN scan. The fixed policy stays the default; the LLM adds readable rationales, not accuracy.
  • Graviton against x86 (same 32-board workload, 3 interleaved repetitions, medians, deployed configuration): the classical path takes 37.8 ms on c8g.xlarge against 56.8 ms on c7i.xlarge (1.5x faster). Per 1,000 boards Graviton costs $0.044 against $0.049 with WideResNet-50 and $0.017 against $0.021 with the ResNet-18 throughput mode.
  • COOL, as a measured comparison only. We subscribed to the COOL Marketplace image (OpenCV 5.1.0-dev with KleidiCV 0.7.0), deployed it on the same c8g.xlarge and benchmarked it against stock OpenCV 5. COOL was faster on CLAHE (1.40x), adaptive thresholding (1.37x) and ORB (1.19x) but slower on Sobel and connected components, and the classical path as a whole ran 8 % slower (41.1 ms against 37.9 ms). So the deployed worker uses stock OpenCV 5, the COOL subscription was cancelled, and we do not enter the COOL award. The COOL variant stays selectable in the stack.

Challenges we ran into

  • Our first design was not good enough, and we kept the numbers. A global patch memory at 256x256 with a zoom-first policy reached AUROC 0.877 on pcb1, and its zoom changed 13 decisions, 9 of them wrongly. Raising the input to 448x336 gave 0.938, and fusing with the classical check (rank correlation 0.32 with the DNN on anomalies, so the two are complementary) gave 0.963. Our hypothesis that registering frames first would help the DNN was refuted (0.814 and 0.825).
  • A dataset confound. In VisA pcb1, 96 % of anomalies but only 32 % of normals are rotated by 180 degrees, so orientation alone scores AUROC 0.82. Within one orientation the fused AUROC is still 0.967, and pcb2 and pcb3 show no such confound.
  • Motion blur on a textured board. The frozen quality gate caught horizontal motion blur on pcb4 only 47.9 % of the time, because the variance of the Laplacian stays high on a busy board. A directional gradient-energy ratio now catches 100 % of motion-blurred captures at 3 % of the image width on all 802 test boards in both axes, with 0 % false rejects on clean boards. Because the gap was found on the held-out stress test, that component is no longer a held-out result, and blur below 1 % of the width remains a documented limit.
  • Scale to zero without breaking the demo. The worker stops when idle, so the proxy shows a self-refreshing "starting" page while it boots (about 34 s), and maintenance jobs set a keep-awake tag so the idle-stop does not stop the worker under a running benchmark.

Failure cases and limitations

  • Melted solder is the hardest class: the agent fails it outright only 6.1 % of the time, and stops 59.2 % of melt boards when escalations are counted (n = 98).
  • Of the 16 verdicts the agent changed against a single pass on pcb2 to pcb4, 10 became right and 6 became wrong.
  • 23 % of held-out boards (138 of 602, 130 of them anomalous) go to a person. That suits a sampling station, not a fully automatic line.
  • One golden board per type is brittle: a good pcb3 board fails live in the video on the classical early exit, and the operator overrides it.
  • Evaluation uses VisA under laboratory lighting; real line images have not been evaluated. About 1.2 s per board on CPU: a real line needs batching or an accelerator.

Accomplishments that we're proud of

  • An honest protocol: one development board, a frozen design, one run on three held-out board types, failure galleries and the wrong decisions reported next to the right ones.
  • An agent whose OpenCV 5 output really changes what it does next: re-capture, switch registration method, skip the DNN, zoom, or escalate.
  • A person approves every reject, and every step is traced on screen, in CloudWatch and in DynamoDB.
  • A live, scale-to-zero deployment on Graviton from one CDK stack, with a measured cost per board.

What we learned

  • Two independent kinds of evidence, fused on the scale of good boards, beat a better single model.
  • An agent is worth its cost when it changes what happens to a board, and it must be measured on the boards it changes, not only on the average.
  • Measure an optimisation before adopting it: COOL sped up individual operations but not our pipeline.

What's next

  • A small set of golden boards or per-region tolerances, to cut false rejects from legitimate variation.
  • A pilot on real line captures with a fixture camera, and per-product tuning of the escalation band.
  • Batching or an accelerator for the DNN step to reach line speed.

Disclosure

  • Built by Rui Codex (solo: Selman Altinay) for this competition. We used AI coding assistants while building it; the agentic workflow judged here is the runtime inspection agent, not the coding assistant.
  • The video narration is a synthetic voice (Kokoro-82M), disclosed in the video description. The team photo is a real photograph. No AI-generated images or video are used; all footage is a real recording of the live endpoint.
  • Data: Amazon VisA, CC BY 4.0 (Zou et al., ECCV 2022). Backbone weights: torchvision ImageNet weights (BSD-3-Clause).

Built With

Share this project:

Updates

Submission history