Inspiration
Retail shelves tell a story that traditional inventory systems cannot always capture.
A product can be recorded as available while its shelf position sits empty. For convenience-store employees and retail shift supervisors, finding these problems often means manually walking aisles, inspecting shelves, and communicating restocking needs.
We wanted to explore a simple question:
What if a store could visually recognize a shelf problem, decide what should happen next, and verify whether the problem was resolved?
That idea became ShelfSignal.
Our goal wasn't to create another computer vision dashboard that highlights objects and stops there. We wanted to connect visual intelligence to an operational workflow that helps people make better decisions.
What it does
ShelfSignal is an agentic visual inspection system designed to help retail teams identify potential shelf-stock gaps and track corrective actions.
The workflow follows five steps:
Establish a reference: Upload an image of a stocked shelf and define the regions to inspect.
Detect changes: Upload a current shelf photograph. OpenCV 5 analyzes visual differences to identify possible missing-product regions.
Decide what happens next: The decision engine evaluates the evidence and determines whether to recommend a restock task, request another image, or report no significant change.
Keep humans in control: A supervisor reviews the evidence before approving an operational task.
Verify the outcome: A follow-up photograph is compared against the reference and previous inspection to determine whether the visual issue appears resolved.
The central innovation is the feedback loop.
ShelfSignal doesn't just identify a potential problem. It uses visual evidence to guide an action and evaluate the result.
The inspection interface is designed to make that process transparent through annotated images, candidate regions, decision explanations, and an execution history.
How we built it
We designed ShelfSignal around three core engineering layers: computer vision, agentic decision-making, and cloud infrastructure.
Computer Vision with OpenCV 5
OpenCV 5 forms the foundation of the inspection pipeline.
Instead of relying entirely on a multimodal language model to guess what is missing from a shelf, the system uses deterministic image-processing techniques to examine differences between reference and inspection images.
The designed pipeline includes:
Image validation and preprocessing
Feature matching and geometric image alignment
Shelf-region analysis
Edge, texture, and appearance comparison
Morphological filtering and candidate-region extraction
Visual annotations and evidence scoring
Follow-up image comparison
This approach is designed to produce inspectable measurements rather than relying exclusively on a black-box prediction.
Agentic Decision-Making
The next layer turns computer vision outputs into operational decisions.
ShelfSignal uses an explicit workflow state machine to determine what action is appropriate based on the evidence.
For example, a poorly aligned photograph should trigger a rescan request rather than an unreliable restock recommendation.
When sufficient evidence indicates a possible shelf gap, the system can propose a task for human review.
An optional OpenAI API integration adds natural-language explanations to the structured evidence, while deterministic safeguards retain control over approval and verification.
AWS Cloud Infrastructure
We designed the cloud architecture around AWS Lambda, Amazon S3, Amazon DynamoDB, and Amazon CloudWatch.
AWS Lambda: Executes the backend and OpenCV inspection workload.
Amazon S3: Stores inspection images and generated visual evidence.
Amazon DynamoDB: Maintains inspection records, task states, and decision events.
Amazon CloudWatch: Provides logging and execution observability.
The application uses FastAPI for its backend services and React with TypeScript for its inspection interface.
AWS SAM defines the deployment infrastructure.
Synthetic Evaluation
To make testing reproducible, we designed a synthetic shelf-image generator with controlled examples of missing products, refilled regions, lighting changes, blur, and alignment failures.
The purpose is to test not just whether the system detects a gap, but also whether it behaves appropriately when the visual evidence is insufficient.
Synthetic evaluation is an engineering starting point, not a replacement for real-world store testing. Challenges we ran into
The most important engineering challenges in ShelfSignal center on the reliability of visual interpretation.
- Distinguishing a missing product from an ordinary visual change
Lighting, shadows, packaging, camera perspective, and shelf arrangements can introduce differences that resemble missing products.
Our approach addresses this through reference-based comparisons, defined inspection regions, geometric alignment, and evidence-quality checks.
- Turning visual analysis into meaningful agent behavior
Detecting a region is different from deciding what to do about it.
We designed the agentic workflow so that vision results influence subsequent actions. The system must distinguish between a strong candidate, insufficient evidence, and a verified visual change.
- Balancing AI flexibility with operational reliability
Language models are useful for explaining findings, but shouldn't be the sole authority for safety-sensitive or operational actions.
We separated deterministic computer vision and policy enforcement from optional language-model reasoning.
- Keeping the system explainable
A supervisor should understand why a restock task is being proposed.
Annotated photographs, evidence scores, state transitions, and decision traces are intended to make the system's behavior auditable.
- Deploying computer vision to the cloud
Running native computer vision dependencies in a containerized serverless environment requires attention to library compatibility, image size, memory, processing latency, and infrastructure configuration.
Our architecture uses an OpenCV 5-compatible container approach to address those requirements.
Accomplishments that we're proud of
The strongest aspect of ShelfSignal is its focus on connecting perception to action.
Rather than building an image classifier with a chatbot attached, we designed a complete workflow centered on real operational decisions.
The accomplishments we set out to demonstrate are:
A focused real-world use case: Retail shelf inspection with a clearly defined operator and manual workflow.
Substantive computer vision: OpenCV-based visual analysis rather than using AI-generated descriptions alone.
Evidence-sensitive agent behavior: Different visual results lead to different operational decisions.
Human-in-the-loop control: Task approval remains with the supervisor.
Visual verification: The system evaluates a follow-up image rather than assuming a recommended action was completed.
Reproducible testing: Controlled synthetic data supports repeatable evaluation and failure analysis.
Explainability: The inspection workflow exposes the evidence behind decisions.
Our guiding principle was simple: Every intelligent recommendation should be connected to evidence that a human can inspect.
What we learned
Building ShelfSignal reinforced an important lesson about computer vision and agentic AI:
Perception is only the beginning of an intelligent system.
A model detecting something interesting doesn't automatically create value. Value emerges when that observation leads to an appropriate action and the result can be checked.
We also learned to design around uncertainty rather than hide it.
In real-world visual environments, images will sometimes be blurry, poorly aligned, obstructed, or ambiguous. A reliable agent needs to recognize when it doesn't have enough evidence and request a better observation.
Another key lesson was understanding when not to use a language model.
OpenCV is better suited to repeatable image geometry and measurable visual changes. Language models can help interpret and communicate results, but deterministic rules provide stronger guarantees for state transitions and human approval requirements.
Finally, reproducibility matters. A useful computer vision demonstration needs measurable tests, documented limitations, and visible failures, not just one carefully selected successful example.
What's next for ShelfSignal
ShelfSignal's next stage is moving from controlled reference-based inspections toward practical retail environments.
Real-world pilot testing
Collect permissioned photographs from actual shelves and evaluate detection accuracy across lighting conditions, camera positions, store layouts, and product categories.
Better product-level understanding
Explore object detection, product recognition, and planogram integration to distinguish empty shelf space from genuinely expected products.
Operational integrations
Connect approved restock tasks to existing inventory management and store task systems.
Mobile inspections
Create a streamlined mobile experience allowing employees to capture, review, and verify shelf conditions directly from a smartphone.
Continuous improvement
Use human-reviewed outcomes and labeled failure cases to refine the detection pipeline while preserving explainability.
Our long-term vision is to make ShelfSignal an accessible visual operations assistant for smaller retailers.
The future of computer vision isn't just seeing what's wrong. It's helping people close the loop between observation, action, and verified improvement.
Built With
- amazon-web-services
- aws-lambda
- aws-sam
- cloudwatch
- codex
- docker
- dynamodb
- fastapi
- numpy
- opencv5
- pillow
- playwright
- pydantic
- python
- react
- s3
- tailwind-css
- vite
Log in or sign up for Devpost to join the conversation.