vision agent

“Budget-aware agentic visual inspection: OpenCV 5 + COOL on AWS Graviton4, with a human in the loop.”

Comment

Inspiration

On a factory line, a camera checks thousands of parts every minute. Most parts are clearly fine, but normal inspection pipelines still run every heavy operation on every part at full resolution. Meanwhile, the hard parts (a faint crack, a tiny blowhole near an edge) get the same quick look as the easy ones.

A human inspector works differently: they glance at the whole tray, then pick up only the doubtful parts for a closer look. I wanted an AI agent that does the same thing, so that a faster OpenCV build (COOL on AWS Graviton4) turns directly into more defects caught at the same speed.

What it does

VisionGuard looks at one camera frame showing a tray of 9 magnetic tiles and marks each tile PASS, FAIL (with the defect outlined) or NEEDS HUMAN.

It is an agent, not a single model:

  1. Perceive: scan all 9 tiles at low resolution with OpenCV 5.
  2. Decide: clear tiles are decided right away. Uncertain tiles go into a queue, riskiest first.
  3. Act: for each uncertain tile, zoom in at full resolution, then try a CLAHE contrast-enhanced view, but only if the tool's measured cost still fits the time budget.
  4. Escalate: a tile that is still ambiguous, or a tool that keeps failing, goes to a person.
  5. Report: return verdicts, defect outlines and a full JSON trace that shows every step, its reason and the budget left.

The agent calls tool $t$ only if

$$\hat{c}t \le B - \sum{k \in \text{done}} c_k$$

where $\hat{c}_t$ is the running mean of that tool's measured runtime on this machine and $B$ is the time budget per tray. So when OpenCV runs faster, the agent can afford more close looks.

The web dashboard lets you:

  • upload or pick a tray;
  • set the budget;
  • inject a tool failure;
  • watch the trace timeline;
  • compare the same tray at 80 ms and 1000 ms side by side.

How we built it

  • Data: the public Magnetic-tile defect dataset (1,344 real photos with pixel masks; Blowhole, Break, Crack, Fray, Uneven). A fixed 60/40 train/test split by file-name hash means test images are never seen in training. I built 50 test trays (450 tiles, 129 defective).
  • OpenCV 5 features (30 per view):
    • a residual image with the streaks removed (blur 1×81, absdiff);
    • dark and bright blobs against a medianBlur background;
    • edge-band statistics;
    • row and column profiles;
    • Sobel, Canny and Laplacian statistics;
    • contours (morphologyEx, findContours, minAreaRect).
  • Scorers: three small gradient-boosted tree models (coarse, zoom, enhance), trained on the train split only. Cross-validated AUROC is 0.914, 0.925 and 0.928. They are exported to JSON and run with plain numpy, so the COOL container needs no extra packages.
  • Agent: a bounded state machine with a step cap, an escalation cap, one retry per tool, fault injection and a full trace.
  • AWS: Amazon ECS on a c8g.large (Graviton4) running the COOL Marketplace container (OpenCV 5.1) unmodified, with CloudWatch logs. /api/info shows exactly which OpenCV build and CPU are loaded.
  • Baseline: stock opencv-python-headless 5.0 on the same instance with the same inputs. The benchmark uses warm-up runs, 60 timed runs with median and p95, CPU and memory measurements, and a pixel-by-pixel correctness check.
  • Tests: 16 automated tests cover these cases:
    • different visual evidence leads to different actions;
    • more budget leads to more steps;
    • a retry recovers from a tool failure;
    • repeated failure leads to escalation;
    • the step cap and escalation cap hold;
    • full real-tray runs and the web API work.

Challenges we ran into

  • Hand-made rules weren't enough. My first threshold-based detector only reached an AUROC of about 0.72, because faint "uneven" bands and dark-edge breaks look like normal texture. Combining 30 OpenCV features in a small learned model raised it to about 0.91.
  • Overfitting hides problems. The scorers were perfect on training trays, so I couldn't tune the human-review cap on them. I set it to 1 per tray as an honest product decision based on operator capacity, and I report it that way.
  • A bug the tests caught. The step cap wasn't checked in the final "locate defect" stage. A test with a cap of 5 saw 12 steps, so I fixed the loop.
  • Running COOL correctly.
    • It runs only on ECS/EKS on Graviton4 (no Fargate).
    • It needs aws-marketplace:RegisterUsage on the task role.
    • It runs as user 1000.
    • It exits with code 133 without an entitlement.

I turned that exit code into proof: if the probe task runs successfully, COOL really ran.

  • Fair benchmarking. Same host, same Python and same images for both builds, with warm-up separated and outputs compared pixel by pixel. Otherwise a "speedup" can come from doing different work.

Accomplishments that we're proud of

  • A real agent, not a chatbot. What the camera sees changes which tool runs next, and the tests prove it.
  • More budget means more defects caught, measured on held-out trays:
Budget per tray Defects caught automatically False alarms Tool calls
60 ms 73.6% 4.4% 1.0
250 ms 84.6% 5.2% 6.2
500 ms 88.0% 6.3% 11.0

With human review included, recall reaches 89.1%. [Add Graviton4 COOL vs stock OpenCV speedup, pixel difference and recall at equal budget here.]

  • Every decision is explainable. Each one comes with its reason, the views used and the budget spent. Quick decisions forced by the budget are flagged, not hidden.
  • Easy to reproduce: pinned versions, deterministic splits, one-command AWS deploy and teardown, 16 passing tests.

What we learned

  • An agent is only as good as its cost model. Planning with measured runtimes, not guesses, is what links library speed to result quality.
  • Honest failure handling (retry, escalate, flag "decided quickly") matters more to a real operator than a slightly higher accuracy number.
  • A benchmark must check correctness, not just speed, and must run both versions in exactly the same conditions.

What's next for vision agent

  • Test on photos of real trays with uneven lighting across the frame.
  • Learn the human-review cap from operator feedback.
  • Set the budget automatically from the production line's cycle time.
  • Add more part types and a camera stream input so it can inspect continuously on the line.

Built With

Share this project:

Updates

Submission history