ShelfSense

Inspiration Small shops (kiranas in India, corner stores everywhere) can't afford Amazon Go-style ceiling rigs, but most already have a security camera pointed at the shelves, often a consumer one like Ring. We wanted to know how much of "just walk out" a single existing camera can do honestly. The key word is honestly: a system that bills people has to know when it isn't sure, and say so.

What it does ShelfSense connects to a shop's Ring camera, or any phone or laptop camera, and watches shelf zones the owner draws on the live feed.

Takes go on the bill. When a shopper's hand goes into a shelf zone empty and comes out holding something, that's a take. The product is named by matching the hand crop against the shop's own photos of the items on that shelf, and a line goes onto that shopper's bill. Grabbing the same item again counts once. Put-backs are caught. A hand that goes in holding something and comes out empty flags the matching line. It never guesses with someone's money. Anything uncertain is marked "verify at exit". The system only ever adds or flags lines. Only a person at the exit check removes one, and every decision is logged. The exit check shows staff the evidence for each flagged line: the hand just before and just after the shelf visit, the scores, and what else it could have been. Staff keep or remove the line, with a note. The shopper's phone. Each session has an unguessable link, shown as a QR code. Shoppers watch their bill fill in live and can tap "Not mine" on anything that isn't theirs. Shop layout and stock. The owner maps the shop (shelves, fridges, freezers, the counter) with products, stock and minimums per shelf, and ties each shelf to a camera zone. Takes lower that shelf's stock, lines removed at the exit put it back, and low shelves are highlighted. Shelf watch. While no hand is in a zone, the same camera reads how full the shelf is (empty, low, half full, full) and raises a restock alert after two low readings in a row. How we built it Ring integration (all real Ring APIs, tested end to end on the Ring Developer Portal Playground):

GET /v1/devices on api.amazonvision.com with the shop's token. Refresh tokens rotate and are saved. Live view over WHEP: our server pulls the video itself with aiortc. It posts the SDP offer, gets 201 with the answer and a Location, and sends DELETE to close. Setup took 0.6–0.9 s, and we got 1280×720 at 29 fps, analysed at about 10 fps live. Ring's 30 s / 60 s live-view cap is handled with three modes: now, on motion, and continuous (a bounded reopen). Signed webhooks: X-Signature = sha256=hex(HMAC-SHA256(key, raw body)), with repeated request IDs ignored. motion_detected opens the live view on that camera. One-way account linking: Token Exchange URL and Account Link URL. The nonce is base64url(HMAC(key, "time:account_id")). Linking is confirmed with POST and PATCH /v1/accounts/me/app-integrations, and app_integration_removed unlinks. The vision pipeline is a funnel, the pattern Ring's Computer Vision Guidelines recommend:

Every frame (13–15 ms): YOLO11n-pose and ByteTrack find people and wrists. Hand boxes sit just past the wrist along the forearm. Only when a hand visits a shelf zone (~200 ms): we take crops of the hand just before it entered and just after it left, and ask Qwen2.5-VL-3B one batched yes/no question: "Is this hand holding a product?" The gap between the Yes and No logits becomes a calibrated probability through vjev, our open-source typed-decision layer. Empty then holding is a take; holding then empty is a put-back. Only on a take (49 ms): SigLIP so400m embeddings are compared with the shop's catalogue photos, restricted to the products on that shelf, with a softmax at a fitted temperature. Guardrails: a confidence below 0.5 means "verify at exit". Repeat takes within 6 s are merged. The system:

A FastAPI server with a session per camera, so the newest frame wins and model work runs off the event loop. Ingest from Ring (WHEP), browsers (WebRTC/WHIP, or a WebSocket JPEG fallback) and single-frame POSTs. Web pages: the console, exit check, shopper's phone, shop layout and cameras. We designed them on a Claude Design canvas first, then built them. Hosted on Amazon EC2: one g4dn.xlarge (NVIDIA T4), the Deep Learning Base AMI, Caddy for HTTPS, and a systemd service, with scripted setup in live/deploy/aws/. 42 automated tests. The demo video was recorded by a Playwright script driving the real UI on the EC2 host, against the live models, and edited with Hyperframes on Kaggle. Evaluations ran on Kaggle GPUs. Challenges we ran into Testing with a real Ring device from India. Staging needs US-located devices and Ring Protection, so everything Ring-side was built and tested on the Playground. The Playground streams the same opening 30 s of its clip on every session, and it sends no webhooks. We tested webhook signatures with our own HMAC-signed events, and demonstrated takes on shelf footage. Hands are small and move fast. Getting the hand crop to land on the hand is the hardest part. On close-up handheld stock clips it often misses, and we report that rather than hide it. Making the system humble. Calibrating the VLM's yes/no so that "unsure" really means unsure, and designing the bill so that a wrong guess can never cost a shopper money. Accomplishments that we're proud of A complete, honest loop: camera → take → bill → shopper's phone → exit check → stock. It runs live on real Ring video on one GPU. Naming: 98.4% correct within a shelf of 5 products (GroceryStoreDataset: 81 products, 500 queries). Shelf watch: AUROC 0.88 on 484 real shelf crops. It's right 82.2% of the time on held-out shelves and catches 92% of empty gaps. The whole Ring connector: devices, WHEP, the live-view cap, signed webhooks and account linking. vjev, released as open source. Its boxes reach 86.3% [email protected] on RefCOCO, and it abstains on 93.8% of absent targets. What we learned Tiered funnels make VLMs affordable: the big model runs only when a hand actually visits a shelf. Matching against the shop's own photos beats asking a VLM for the product's name. It's 6× faster and more accurate. Calibration and abstention matter more than raw accuracy when money is involved. Server-side WHEP is very pleasant to build on. Following the spec made a Python client work on the first try. What's next for ShelfSense A fixed-camera test set recorded in a real shop, to measure and improve take finding where it's weakest. A pilot with one kirana store. Multiple cameras per shop, with shoppers followed across cameras. Mapping camera zones onto the floor plan automatically. Restock alerts pushed to the owner's phone.