ShelfWatch Agent

Synthetic example: three likely gaps and one uncertain cell requiring review.
Generated stocked reference used for shelf comparison.
Generated test image with three removed items and one partially occluded item.
ShelfWatch running on Render with visual evidence, agent decisions and human approval
ex3
ex4
ex1
ex2

Inspiration Small neighbourhood shops need a practical way to notice shelf gaps without adding another complicated system to their daily work. I wanted to explore a simple approach: compare a new phone photo with a stocked reference, highlight changes, and leave the final decision with the shopkeeper.

That became ShelfWatch Agent—a visual inspection prototype that shows its evidence and asks for help when it cannot make a reliable comparison. What it does ShelfWatch Agent compares reference and current shelf photos, identifies likely removal regions, reinspects changed cells, and prepares a restock draft for review.

Its perception → decision → action loop works as follows:

Perceive: align the photos and locate visual changes within the selected shelf area.
Decide: inspect changed regions more closely, flag uncertain cells, or request a retake when alignment fails.
Act: prepare a draft listing affected locations and wait for human approval. The interface displays the annotated image, cell results, and decision trace. Approval is recorded locally; no supplier is contacted and no order is sent. How I built it
Computer vision: OpenCV 5 uses ORB features and a RANSAC homography to align the images. Lab-colour differences, texture masks, morphology, and connected components identify local changes. Edge and texture comparisons, together with finer cell inspection, help distinguish likely gaps from uncertain changes.
Backend and interface: Python and FastAPI serve a lightweight HTML, CSS, and JavaScript application with image previews, shelf-area selection, inspection results, and approval controls.
Storage: SQLite records drafts and approval events. Uploaded photos are processed in memory rather than saved.
Deployment: The Docker application is live on Render. An EC2-based AWS deployment is prepared, but AWS customer verification remains unresolved, so I cannot claim completed AWS delivery.
Evaluation: A separate 33-case generated stress suite passed 32 cases, with cell-level precision of 94.7%, recall of 100%, and F1 of 97.3%. These are synthetic results, not real-store accuracy measurements. Two credited real-photo development pairs also produced the expected affected location after improvements. Challenges I ran into The first version compared large fixed cells and missed small removals. Moving to local change regions helped detect changes that were diluted by whole-cell similarity.

Smooth backgrounds also produced misleading comparisons. A reference-texture mask reduced those errors.

Occlusion remains difficult: a partially hidden item can resemble a removal. One stress-test case still produces a candidate gap instead of an uncertain result, which reinforces the need for human review. Accomplishments that I'm proud of

Building an end-to-end application that runs OpenCV 5 analysis on a hosted service.
Making visual evidence influence reinspection, retake requests, and approval decisions.
Recording approvals while keeping supplier actions outside the prototype.
Documenting failures and separating generated tests from real-photo development checks. What I learned Alignment quality and detection confidence answer different questions. A well-aligned image can still contain an ambiguous change.

I also learned why testing beyond the simplest demonstration matters. Smaller removals, background changes, and partial occlusion exposed weaknesses that the original examples did not reveal. What's next for ShelfWatch Agent My priorities are resolving AWS verification, completing deployment, and collecting labelled store photos for a separate development and untouched test set.

After that, I want to explore product identification, multiple shelves, and approved restock notifications. Any external action would remain behind an explicit human approval step.