ShelfSentry AI

ShelfSentry AI active-perception shelf inspection that looks again before it concludes.

Comment

Inspiration

Retail shelves go empty, and the usual computer-vision answer is a one-shot detector: it returns a box and a confidence number and stops. That bothered me, because a real store associate doesn't work that way. When a shelf looks empty, a person leans in, checks whether it's actually a gap or just shadow, a divider, or a product pushed back, and only then decides. I wanted to build perception that behaves like that - a system that treats an uncertain finding as a reason to look again, not a reason to guess. And I wanted the vision to be real classical CV doing measurable work, not an LLM hallucinating boxes.

What it does

ShelfSentry AI inspects a retail shelf image (or short video) and decides what's happening through an active-perception loop:

  • Real OpenCV 5 analysis finds candidate empty regions with measured metrics (empty-space ratio) and a quantified uncertainty for each region.
  • A bounded agent (Amazon Bedrock as the primary decision model) reads those findings and chooses the next action: zoom into an uncertain region, analyze another frame, ask for a better image, escalate to a human, or finalize.
  • The agent reaches perception only through four restricted MCP tools no filesystem, URL, shell, or credential surface.
  • Every decision and tool call is recorded in an ordered decision trace, and the final report cites it.
  • It's honest by construction: heuristic scores are kept separate from probabilities, every empty-looking region carries a visual caveat, and it never asserts "out of stock" as proven.

It runs in two modes from the same code: a local/offline pipeline, and an AWS mode (S3 + DynamoDB + Lambda + API Gateway + Bedrock).

Example results (real output from a live run)

A sample inspection on the deployed stack produced the annotated output image plus three measured candidate regions. The values below are real measured output, not illustrative placeholders:

Region Empty-space ratio (estimate) Uncertainty value Heuristic score
region_0000 77.8% 0.428 0.654
region_0001 18.9% 0.539 0.340
region_0002 22.3% 0.446 0.354

Each region carries a visual caveat, e.g. "This region is an estimate of empty shelf space, not a confirmed out-of-stock condition. An empty-looking region may instead be shadow, occlusion, shelf dividers, or poor image quality." The empty-space ratio is a measured value, the heuristic score is an ordinal (not a probability), and the uncertainty value has a documented interpretation - the system never asserts out-of-stock as proven.

How I built it

  • Vision: a fixed, deterministic OpenCV 5 pipeline (cv2 5.0.0) Sobel/Hough shelf-band detection, Otsu + adaptive thresholding, morphology, Canny edge density, and contour analysis producing typed findings with uncertainty.
  • Agent: a bounded Strands-style loop (max 4 iterations) with Amazon Bedrock (Nova Lite via the Converse API) as the primary decision path and a deterministic policy as a recorded fallback.
  • Tool boundary: a restricted MCP server exposing exactly four vision tools.
  • Contracts: Pydantic v2 typed models at every seam, versioned response envelopes.
  • Frontend: React + TypeScript + Vite + Tailwind, with a typed API client.
  • Infrastructure: a single CloudFormation template (no CDK/Terraform) deploying a container-image Lambda (OpenCV 5 doesn't fit a zip), API Gateway HTTP API, DynamoDB, a private encrypted S3 bucket, least-privilege IAM, and CloudWatch.

Challenges I ran into

  • OpenCV 5 won't fit in a Lambda zip. Its native libraries exceed the 250 MB unzipped limit, so I moved to a container-image Lambda (ARM64/Graviton) built and pushed to ECR and made the Docker build fail if cv2.__version__ isn't 5.x.
  • Model access reality. The target account didn't have Claude enabled, so I switched the decision model to Amazon Nova Lite. Because the agent uses the model-agnostic Converse API, that was a config change, not a code change.
  • Live deploy surfaced real bugs which I fixed, not hid. CloudWatch logs caught a DynamoDB key-schema mismatch (table keyed on the wrong attribute), a missing bedrock:InvokeModel permission on the API role, a multipart upload parser that mangled binary PNG bytes, and a KeyError when the agent legitimately chose request_better_image. Each was diagnosed from logs and corrected.
  • Keeping it honest under pressure. It would have been easy to fake a vision result or a metric. I kept the recorded-fallback design so the trace always shows why a decision was made.

Accomplishments that I'm proud of

  • A genuinely agentic loop where OpenCV uncertainty causally drives the next action, verified live on AWS Nova Lite chose inspect_crop, the MCP tool ran a real crop, and the agent then finalized.
  • Real, measured evaluation surfaced in the product (not fabricated): 8 synthetic samples, decision correctness 1.0 over the labeled subset, with the 2 false negatives and all limitations reported openly.
  • One template, full stack, least-privilege, reaching CREATE_COMPLETE/UPDATE_COMPLETE with a working public API endpoint.
  • A hard honesty line held throughout: no fabricated confidence, caveats on every empty region, measured-only eval records.

What I learned

  • Active perception is a design discipline, not a feature the value is in when the system refuses to conclude and gathers more evidence.
  • "It built" should mean something: baking the OpenCV 5 version assertion into the image turned a possible silent regression into a loud build failure.
  • The Converse API's model-agnosticism is a real portability win swapping Claude → Nova was trivial.
  • Honest fallbacks beat confident guesses. Seeing used_fallback: true with a reason in the trace is more trustworthy than a smooth but unexplained answer.

What's next for ShelfSentry AI

  • Tighten the Nova prompt to reduce occasional non-JSON responses (which currently trigger the recorded fallback).
  • Replace the history Scan with a GSI-backed query for scale.
  • Move long-video processing to ECS/Fargate when a job can exceed Lambda's 15-minute limit (the container image already supports it).
  • Evaluate against real shelf imagery with human-labeled ground truth, so numbers reflect production accuracy rather than synthetic labels.
  • Add annotated-evidence serving in AWS mode and a reviewer workflow for the needs_review outcomes.

Built With

Share this project:

Updates

Submission history