One workspace for original images, detection methods, and measured results.
Inspect detection precision, recall, counting errors, and performance across image difficulty levels.
Compare orange model predictions with blue reference annotations. RPC imagery: Wei et al., CC BY-NC-SA 4.0.
Compare completed methods on identical inputs using detection, SKU-matching, and counting metrics.
Follow how OpenCV observations select the next tool, with recorded decisions, timings, and human-inspection recommendations.
Inspect original retail footage and frame-level diagnostics. Footage: Standard Cognition / RetailAction, Standard.AI Dataset License.
Measured COOL versus stock OpenCV performance on AWS Graviton, with matching evaluation scores and downloadable evidence.
CheckoutCheck architecture: AWS hosting, OpenCV 5 processing, agent decisions, evidence storage, and COOL benchmarking.
From image to insight: verify original inputs, analyze with OpenCV, select the next tool, and inspect measured results with human oversight.

Inspiration

A checkout alert affects someone standing on the other side of the camera.

When a system misses an item, a retailer may lose money. When it raises an unnecessary alert, a customer may face an uncomfortable interaction. Accuracy matters, and so does understanding the evidence behind a decision.

We built CheckoutCheck to help teams answer a practical question: what did the camera observe, what did the model conclude, and where did it go wrong?

Our goal is to give developers, researchers, and retail teams a useful workspace for comparing models, investigating failures, and keeping human judgment involved.

What it does

CheckoutCheck is a visual evaluation application for camera-assisted checkout systems. It brings original images, retail footage, model predictions, and measured results into one workspace.

Users can:

Compare detection methods on identical photographs using precision, recall, counting errors, and results by difficulty.
Inspect predictions by toggling model boxes and reference annotations to identify detected, missed, and incorrectly identified products.
Review real retail footage with playback, frame seeking, source timestamps, and image-quality diagnostics.
Follow agent decisions as OpenCV observations determine which tool runs next and when human inspection is recommended.
Export reproducible reports containing configurations, hashes, predictions, and scores.
Inspect cloud performance through our COOL-versus-stock OpenCV benchmark on AWS Graviton.

The public application lets visitors explore completed evaluations, compare results, and inspect supporting evidence. The private operator workspace handles uploads and new evaluation jobs.

CheckoutCheck evaluates visual observations. Its current image and video measurements do not establish whether a customer scanned or paid for an item.

How we built it

We built the browser interface with JavaScript, HTML, and CSS, backed by Python, FastAPI, and SQLite. A background worker manages evaluation jobs, cancellation, and persistent results.

OpenCV 5 performs core vision work: image decoding, preprocessing, foreground segmentation, watershed separation, SIFT feature matching, and video diagnostics. PyTorch and torchvision run the neural detectors.

We evaluated 24 original RPC validation photographs containing 291 annotated products. A separate set of 400 original training photographs supported reference matching and detector fine-tuning. We also integrated six original RetailAction clips for video inspection.

Our vision agent follows a bounded, deterministic workflow. It measures image sharpness and foreground structure, selects a neural detector or watershed refinement, and records the observations behind that decision. Failed follow-up tools and significant count disagreements lead to a recommendation for human inspection. Each image has a maximum budget of three tool calls, and the agent does not automatically intervene in transactions.

AWS EC2 runs the application and evaluation workloads. Private S3 stores approved inputs and model weights, while evaluation results remain on EC2. CloudFront provides HTTPS access to a separate read-only review service.

For the COOL evaluation, we ran the application's foreground and watershed pipelines using the official Marketplace runtime on AWS Graviton and compared them with stock OpenCV on the same instance.

Challenges we ran into

One of our biggest challenges was connecting useful visual measurements to the evidence required for checkout decisions.

Real retail photographs and videos were available, but synchronized checkout footage and recorded scan logs were not. We built and evaluated the image and video workflows while keeping transaction-level validation as a separate requirement.

The photographs exposed difficult detection cases: touching products merged together, packaging resembled the background, and occlusion made matching unreliable. Our trained detector improved recall over the other baselines, but still missed many products. We made these failures visible through overlays and per-image results.

The agent presented another challenge. Its adaptive workflow produced inspectable decisions, but its detection recall was lower than running the trained detector alone. Keeping that comparison visible helped us evaluate the value and tradeoffs of orchestration.

Cloud deployment introduced worker timeouts and the need to expose results safely. We addressed these with bounded execution and a separate read-only service using a consistent database snapshot.

Accomplishments that we're proud of

We built a working application where someone can open a result, inspect the original input, compare methods, and understand how a score was calculated.

On our exploratory evaluation set, the RPC-trained detector achieved 74.47% precision and 36.08% recall. It still missed 186 of 291 annotated products, and the application exposes those misses alongside false detections.

The agent completed all 24 images, selected the neural path 18 times and watershed six times, and recommended human inspection for 17 images. It stayed within its three-tool-call budget for every image.

Our COOL benchmark measured 1.033× and 1.041× observed speedups for the foreground and watershed pipelines, respectively, with identical evaluation scores. We documented the small sample, sequential run order, and runtime-version differences so readers can assess the results in context.

We also passed 71 automated tests, verified application workflows in the browser, and deployed a public AWS review application. The source is available for the community to inspect, reproduce, and improve.

What we learned

A useful evaluation application makes failures as easy to inspect as successful predictions.

A single accuracy number cannot explain every missed object, unsuitable input, or uncertain decision. Original images, overlays, per-image results, and reproducible reports give teams the context needed to choose their next improvement.

We also learned to evaluate an agent's workflow separately from its prediction quality. Completing every step within a tool budget demonstrates execution reliability; detection quality still needs its own measurements.

Seeing a product, observing an interaction, and confirming a transaction each require different evidence. Keeping those distinctions clear has shaped both the application and our plans for its next release.

What's next for CheckoutCheck

Our next priority is a consented checkout pilot with synchronized video, recorded scan events, and independently reviewed transaction outcomes. That will let us evaluate the complete workflow against actual checkout evidence.

We plan to:

Improve detection and tracking across lighting, occlusion, packaging, and camera positions.
Evaluate on a larger, untouched test set.
Calibrate uncertainty and measure when human review is useful.
Add reviewer corrections with a clear history of annotation and decision changes.
Develop the access, retention, and operational controls needed for a broader pilot.

We want CheckoutCheck to help the community build checkout vision systems that people can inspect, question, and improve.