VisionProof

• 

• 

• 

"Premature optimization is the root of all evil." Donald Knuth

Unverified optimization is worse. We made a detector 2.8x faster, and it quietly stopped seeing 4 of 15 people. VisionProof caught it and blocked the change.

The problem: faster can mean blind

Every vision team eventually speeds up a model: smaller inputs, lower resolution, more threads, cheaper hardware. The speedup shows up instantly in a benchmark. The objects it stopped detecting don't.

What we measured while building VisionProof:

• Pedestrians: shrinking YOLOX input from 320 to 160 px made it about 2.8x faster, and recall fell from 100% to 73.3%. Four people the baseline found were simply gone.
• Circuit boards: a PCB setting change that was only 1.04x faster dropped recall from 86.4% to 24.5%, losing 559 defects.
• Even "obvious" optimizations backfire: an AI-proposed downscale to 0.75 was slower (662 ms vs 460 ms per batch) and lost 79 detections.

A latency chart shows none of this. That is how quality regressions ship.

Where it matters:

• Factory inspection: a weld or PCB line downscales images to keep up with the conveyor, and small defects slip through.
• Safety and robotics: a pedestrian detector on a forklift, robot or traffic camera is tuned for frame rate and misses people at the edge of the frame.
• Retail and warehouses: counting cameras move to cheaper hardware and quietly undercount.
• Cloud cost cuts: moving inference from x86 to Arm saves money, but needs proof that detections did not change.

What VisionProof does

VisionProof answers one question before you adopt an optimization: is it faster without losing what it used to find?

1. Same images, two configurations: your current setting versus the faster candidate.
2. Limits fixed before the run: recall, precision and latency, so speed can never override a quality loss.
3. Every lost detection shown: missed objects are drawn in red, side by side.
4. A verdict with evidence: pass or blocked, plus a downloadable report of the contract, raw timing blocks and lost detections.
5. In CI: vision-proof check runs in GitHub Actions on pull requests (exit 0 pass, 2 regression, 3 error).

In the demo, the 160 px candidate was blocked (4 detections lost). Keeping 320 px and adding 4 batch workers instead passed with 0 lost. Same goal, very different outcome.

How we built it

• OpenCV 5 does the vision work: the OpenCV Zoo YOLOX person detector through the DNN module on the Penn-Fudan pedestrian dataset, and a classical PCB defect locator on DeepPCB. Both share one scorer (one-to-one matching at IoU > 0.33), one quality contract and one evidence format.
• Amazon Bedrock (Nova 2 Lite) is the experiment planner. Through one propose_experiment tool it suggests a bounded setting, reads the measured result, and proposes the next. It cannot touch the evaluator: the model suggests, it never grades itself.
• AWS Graviton and COOL: the same evaluator ran on x86 (c7i.large), Graviton (c8g.large) and Graviton with COOL from AWS Marketplace. All three passed with identical detections; COOL recorded about 4% lower warm batch latency than pip on the same Graviton instance.
• Browser workbench: presentation view, live webcam comparison (prediction differences only, never claimed as accuracy), dataset import, and the full experiment trace including rejected ideas.

Results

• Bedrock trace (PCB): 4 proposals, 2 rejected. The selected setting was 3.7x faster than baseline on held-out test images and about equal (0.99x) to a simple fixed sweep. The agent found a good answer, not a better one than the sweep.
• Recorded CI study: the 1.04x change that lost 559 defects was blocked.
• GitHub Actions on PR #1: both pipeline checks passed and uploaded their reports.

What's next: from one check to every vision release

• Every model change, not just settings: model versions, quantized or pruned models, and new OpenCV releases get the same pass or fail answer with lost detections attached.
• Edge and fleet rollouts: qualify a setting once per hardware type (Graviton, x86, edge boards) before it reaches every camera or robot.
• Cost-aware optimization: check many candidates in parallel on short-lived AWS workers and keep the cheapest instance that still passes. Graviton and COOL make that search cheaper.
• Agents that optimize safely: let the Bedrock planner explore far more settings while the independent gate guarantees nothing skips the quality check.
• Existing workflows: common annotation formats, report exports to dashboards, and a required check on pull requests across many repositories.

No vision team should ship a "faster" model without seeing what it stopped detecting.

Thank you to the sponsors

OpenCV 5 is the reason this was buildable by one person. The DNN module runs OpenCV Zoo models like YOLOX with no extra framework, so pushing the exact same pixels through two configurations took a few lines of code. The classical toolkit (registration, thresholding, morphology, contours) powered the whole PCB pipeline. One library covered both a neural detector and a classical inspection line, which is exactly what made a single shared quality gate possible.

AWS turned a local tool into a real system:

• Amazon Bedrock made the agent practical. Tool use with Nova 2 Lite gave us structured, bounded proposals instead of free text, with token usage recorded for every call.
• AWS Graviton gave us a second architecture to qualify on, at a lower hourly rate than the comparable x86 instance we tested.
• COOL on AWS Marketplace let us run the same OpenCV workload as a ready-made Arm build, and it matched pip's detections exactly while recording lower latency.
• EC2 let us spin up short-lived workers, run the identical evaluator, and terminate them, so cloud results are reproducible evidence rather than screenshots.

How we would involve the sponsors more

• COOL as the default runtime for qualification. Run every VisionProof check on Graviton with COOL, and publish the per-change speed and quality report as a standard artifact.
• Bedrock as a smarter planner. Give the agent more budget, compare Nova models against each other, and use Bedrock Guardrails and human approval steps before any proposal is promoted.
• Serverless and container checks. Package the evaluator for AWS Lambda or ECS so every pull request in a large organization can be qualified in parallel without managing servers.
• More OpenCV 5 models. Add OpenCV Zoo face, text and segmentation models so any OpenCV team can gate their own pipeline.

What we would love to see from OpenCV and AWS

• Matched COOL and pip builds. The COOL image shipped OpenCV 5.1.0-dev and NumPy 2.5.1, while pip gave 5.0.0 and 2.4.2, so we could not isolate COOL's own speedup. A matched pip baseline or a published version table would make fair comparisons easy.
• Smoother Marketplace onboarding. Our first COOL subscription attempt hit a Marketplace payment error. A short troubleshooting note or a trial path for competitors would save time.
• An official benchmarking protocol. Recommended warm-up, repeats and randomized order for COOL versus pip would help every team report comparable numbers.
• A built-in quality check in OpenCV Zoo. Shipping small labeled sample sets with each Zoo model would let any developer verify that an optimization keeps accuracy, the same check VisionProof runs.

Limitations and responsible use

The four-image imported set is illustrative and its labels are user-supplied. Single-image timings are not batch results or webcam frame rates. CI compares configuration changes against the current implementation, not arbitrary code revisions, and does not enforce branch protection. Tools like Roboflow, FiftyOne and Ultralytics offer related evaluation; VisionProof focuses on gating one optimization with its lost detections in plain view. Camera frames are processed locally and never saved.

Technical write-up: architecture page, README.md and DESIGN.md in the repository.

Built With

• amazon-bedrock
• amazon-ec2
• aws-graviton
• cool
• github-actions
• numpy
• opencv
• python
• vercel
• yolox