Inspiration
WEB&SHOP grew from a practical question: can a shop use product photographs to make inventory work easier while keeping people in control? Product information often lives across spreadsheets, shelf labels and stock records. I wanted owners, employees and buyers to work from one shared catalogue, connecting visual recognition with useful actions.
What it does
Owners register a product using 3–5 reference photographs, review image quality, and enter its name, price, opening quantity and shelf. Registration saves a unique SKU and reference embeddings without training a new model for that product. Employees upload a different photograph and receive product boxes, suggested SKUs and unknowns. They correct assignments and quantities before approving an ADD, SUBTRACT or shelf-audit action. Stock changes only after explicit confirmation, and updated Excel exports reflect the confirmed balances. Buyers use the same catalogue to build shopping lists, calculate totals, check a budget and find products on a simple 2D shelf map. Buyer previews do not change inventory.
How I built it
The application uses Python, Streamlit and SQLite, with three distinct computer-vision components:
- OpenCV 5 on AWS Lambda, triggered through private S3 uploads, checks image sharpness, exposure and resolution.
- NVIDIA Retail DINO on a RunPod GPU proposes product bounding boxes.
- Pretrained SigLIP2 running locally compares detected crops with the selected shop’s registered references. I fine-tuned the generic DINO product detector using NVIDIA TAO Toolkit 5.5. Reviewed photographs from the first two collections supplied 25 training images with 144 boxes, with five images and 33 boxes reserved for validation. Training ran for 12 epochs. A third collection supplied separate evaluation scenes. Its owner reference photographs supported reference matching rather than detector training. The final application reused the trained detector with the grocery-shop catalogue and newly registered references.
Challenges I ran into
The hardest problem was separating physical products reliably. Touching packages, partial products and repeated items can produce missed, merged or overlapping boxes. A confident-looking SKU suggestion can also be wrong. I made correction and approval central to the workflow: employees can adjust quantities, correct unknowns, remove unwanted proposals and add missed products. Inventory actions include before-and-after previews and duplicate-posting protection. Another challenge was keeping opening quantities, current stock, shelf balances and Excel exports consistent. Downloaded spreadsheets are snapshots, so users need a fresh export after approved changes. Limited local disk space and GPU costs also shaped the project. The architecture keeps the catalogue and buyer tools available locally while GPU inference can be enabled for scheduled demonstrations.
Accomplishments that I'm proud of
The demonstration connects a newly registered product to employee recognition, an explicitly approved stock change, updated Excel balances and the buyer catalogue. It shows one complete workflow with real inference and visible human corrections, using fictional shop data.
What I learned
I learned that detection, recognition and inventory correctness are different measurements. On 30 reviewed evaluation scenes, DINO achieved 74.5% precision and 71.1% recall for product localization. Separately, SigLIP2 achieved 93.4% top-one candidate-group agreement on 290 reviewed, manually cropped items. Reviewers saw model suggestions, so this is not a blind recognition test or end-to-end inventory accuracy. Keeping evaluation photographs outside detector training helped expose weaknesses. However, those scenes had already been inspected during development, so a larger independent test is still needed. Most importantly, useful AI does not have to hide uncertainty. Showing unknowns and making corrections straightforward can be more valuable than presenting every prediction as certain.
What's next for WEB&SHOP
I want to evaluate more independent shelf arrangements, measure unknown-product rejection and correction time, and improve crowded-scene detection. Further work would also address durable hosted storage and simpler deployment. The goal remains the same: turn product photos into reviewed stock updates and smarter shopping.
Log in or sign up for Devpost to join the conversation.