Inspiration

Retail shelves can change quickly throughout the day. Products may be missing, misplaced, or arranged differently from their expected positions, making manual shelf audits time-consuming. We wanted to explore how computer vision could make this process faster and more transparent.

This inspired us to build ShelfSherlock, an agentic visual inventory auditor that compares shelf images, investigates visual differences, and helps identify changes that may require attention.

What it does

ShelfSherlock analyzes a reference shelf image and a current shelf image to identify and visualize potential changes.

Key capabilities include:

Visual comparison: Compares two shelf images to identify differences.
OpenCV-powered analysis: Uses image processing, alignment, and difference detection to highlight changed regions.
Agentic investigation: Uses analysis results to decide what to do next, such as inspecting a region more closely, retrying alignment, or requesting human review.
Evidence-backed results: Presents highlighted image regions, analysis metrics, and an investigation trace to help users understand the outcome.
Human-in-the-loop review: Flags uncertain findings rather than treating every visual difference as a confirmed inventory issue.

ShelfSherlock is designed to support inventory audits, not to claim exact stock counts from images alone.

How we built it

We designed ShelfSherlock around three main components:

Computer vision: OpenCV 5 is intended to handle image preprocessing, alignment, difference maps, and changed-region extraction.
Agentic decision loop: A Python backend uses analysis results to determine the next investigation step and records the actions taken.
Interactive web interface: A React frontend allows users to provide shelf images and review visual evidence, results, and investigation steps.

The backend is designed with FastAPI, with API endpoints connecting the frontend to the image-analysis pipeline. AWS integration is planned as the cloud component for the final deployment.

Our focus is on making the investigation process observable: users should be able to understand not only what changed, but also how the system reached its result.

Challenges we ran into

One major challenge is distinguishing meaningful shelf changes from differences caused by lighting, camera angle, image alignment, and other visual noise.

Another challenge is making the system genuinely agentic. A useful agent should adapt its next action based on actual analysis results rather than simply follow a fixed sequence of steps.

We also considered the importance of handling uncertain results responsibly. A highlighted region does not automatically prove that a product is out of stock, so human review and clear evidence are important parts of the design.

Finally, we aimed to balance computer-vision accuracy, an understandable interface, reliable testing, and practical deployment constraints.

Accomplishments that we're proud of

We are proud of the concept of turning shelf-image comparison into an evidence-driven investigation workflow rather than presenting a simple difference image.

ShelfSherlock brings together computer vision, adaptive decision-making, visual evidence, and human oversight in one focused application.

We are also proud of the emphasis on transparency: the goal is for users to inspect the evidence and investigation trace instead of blindly trusting an automated conclusion.

What we learned

This project reinforced that computer vision is not just about detecting differences. Reliable results also depend on image quality, alignment, validation, and careful interpretation.

We learned that an agentic system needs a meaningful feedback loop: its observations should influence its next action, and its decisions should be traceable.

We also learned the importance of evaluating failure cases, communicating uncertainty, and keeping humans involved when automated results are inconclusive.

What's next for ShelfSherlock: Agentic Visual Inventory Auditor

Our next steps are to strengthen the system through:

More robust shelf-image alignment and change detection.
Testing on diverse shelf images and challenging real-world conditions.
Better region-level investigation and confidence-based human review.
Evaluation against labeled examples, including false positives and missed changes.
AWS deployment for a reproducible, accessible demonstration.
Clearer audit reports that help retailers prioritize shelf areas for inspection.
Further development of the agent's decision policy so each next action is justified by the evidence gathered.

Our long-term vision is to make ShelfSherlock a practical visual auditing assistant that helps retail teams investigate shelf changes faster while keeping the process transparent, measurable, and human-controlled.