PanelScan Agent: agentic visual inspection on OpenCV 5

• 

  Degraded frame under the agentic loop — 4/4 fiducials recovered, all four defect classes located

• 

  Normal-illumination frame — 0.975 precision / 0.952 recall, no re-capture needed

• 

  Same board, second degradation: exposure and blur measured before any verdict is given

• 

• 

  Low light + tilt: fiducial count drives a reposition instead of a false reject

Inspiration

A single-shot vision pipeline fails in a boring, expensive way: the part is fine, the photo is not. One bad exposure or a slightly tilted fixture and the inspector either rejects a good panel or — much worse — quietly passes a bad one.

Commercial machine-vision rigs fix this by closing the loop: measure the image quality first, then move the sensor. That logic usually ships inside proprietary hardware drivers. PanelScan Agent is that loop rebuilt with OpenCV 5 and no special hardware: the visual evidence itself decides the next sensor action.

What it does

1. Register. Four ArUco fiducials give a homography; the capture is rectified into canonical panel coordinates, so the inspector stops caring how the part was staged.
2. Verify parts where they belong. Every expected screw and label is scored inside its own neighbourhood with normalised cross-correlation against the clean reference board (brightness/contrast invariant, so an exposure change is not mistaken for a defect).
3. Find free-form defects. The rest of the interior is differenced against the reference, thresholded with Otsu, and each connected component is classified by aspect ratio, area and peak residual into scratch / solder blob.
4. Act on the evidence. Fewer than four fiducials → reposition. Laplacian variance below 140 → refocus. Frame too dark → raise exposure. Detection unconvincing → re-capture at higher quality and re-measure. The agent returns the verdict and the decision trace that produced it.

Step 4 is the whole point. This is not a chatbot bolted onto a vision result: this is a vision loop whose sensor actions are conditioned on measured image statistics.

How we built it

• src/perception.py — the OpenCV 5 stage: cv2.aruco marker detection, cv2.findHomography, cv2.warpPerspective, cv2.matchTemplate (TM_CCOEFF_NORMED) for the part checks, cv2.threshold(..., cv2.THRESH_OTSU) on the residual, cv2.morphologyEx, cv2.connectedComponentsWithStats, cv2.Laplacian for focus.
• src/scene_sim.py — a deterministic synthetic bench with ground truth. It renders the same board under controlled degradations (exposure, blur, tilt, occlusion) so every claim below is measurable and rerunnable, and no third-party imagery is redistributed.
• src/agent.py — the perceive → decide → act loop, evidence merging across sensing steps, and the decision trace.
• src/eval_harness.py — agentic loop vs a single-shot baseline, with per-defect-type precision/recall/F1 and latency.
• src/service.py — FastAPI endpoints (/inspect, /inspect/scene/{seed}, /metrics) so the same loop can be driven over HTTP.

Challenges

• Exposure is not a defect. Raw pixel differencing reports every brightness change as damage. Scoring each part inside its own neighbourhood with NCC, and thresholding only the remaining interior, was what separated "the light moved" from "the part is missing".
• Deciding when to stop. An agent that can always ask for a better image will ask forever. The loop is capped and each action must be justified by a measured statistic (fiducial count, Laplacian variance, mean luminance), which also makes the trace auditable after the fact.
• Honest evaluation. The bench is synthetic, so the numbers measure the loop, not a real factory. We kept the failure cases in the repository instead of tuning them away: the scratch class is where precision drops (0.73 on degraded frames) because thin low-contrast scratches land near the Otsu boundary.

Accomplishments that we are proud of

Measured on 40 normal + 40 degraded scenes (OpenCV 5.0.0, CPU only, reproducible with make eval):

The loop buys +0.482 recall on frames that a single shot cannot read, for 3.0× latency and 0.056 precision. Missing screws and misaligned labels go to F1 1.00; the residual cost is concentrated in the scratch class. Per-defect tables and the failure cases are in results/EVAL_SUMMARY.md and docs/REPORT.md.

What we learned

• A measured statistic is a better control signal than a threshold on the final confidence: Laplacian variance and fiducial count are cheap, interpretable, and they generalize across the degradation types we simulated.
• Merging evidence across sensing steps needs a policy, not just a sum — a second look should be able to withdraw a detection, not only add one.
• Reporting the latency cost of the loop (3.0×) next to the recall gain makes the trade-off arguable instead of magical.

What's next

• Replace the synthetic bench with a real staged rig and a calibrated light dome; the degradation model is the first thing that would need re-fitting.
• Learn the action policy instead of the current rule cascade, keeping the measured statistics as the observation vector.
• Run the container on AWS Graviton and compare per-inspection cost against the CPU p50 above.

Honest status — AWS and scope

No AWS deployment is claimed. deploy/ contains a Graviton (linux/arm64) container recipe, an ECS task definition and an ECR push script, and docs/REPORT.md §6 explains how the four stages map onto ECS/ECR — but this entry has no AWS account attached, no AWS credentials were used, and nothing is claimed to be running in the cloud. The "cloud delivery" part of the rubric is therefore only partly addressed: the porting path is written and reviewable, not executed. We also do not claim the Best Use of COOL award — COOL was never executed.

The system is a decision-support inspector, not an autonomous safety authority: it reports a verdict plus the evidence that produced it, it never actuates a production line, and a human remains in the loop for any reject/accept decision on real hardware.

Built With

• aruco
• docker
• fastapi
• numpy
• opencv
• pytest
• python
• uvicorn