Aisle Agent

• 

  Every corner shop already has a camera. Aisle Agent makes it the checkout.

• 

  Measured on the same items: hosted Nemotron VL vs a local 3B VLM.

• 

  Takes at the shelf go on the bill; anything uncertain says verify at exit. The shopper's phone shows the live bill.

• 

  Exit check: Nemotron-3-Super explains the evidence and recommends keep, remove or ask. A person decides.

• 

  Model trace: every Nemotron call on Token Factory, with its latency, tokens and cost.

Inspiration

Small shops can't afford Amazon Go-style ceiling rigs, but almost all of them already have a camera over the shelves. Aisle Agent turns that camera into a checkout and a stock clerk.

What it does

• Takes go on the bill. A hand goes into a shelf zone empty and comes out holding something: that's a take. Nemotron-3.5-Super-VL checks the hand just before and just after the visit, names the product among the ones stocked on that shelf, and a line goes onto the shopper's bill. Stock on that shelf goes down by one.
• Put-backs are caught. A hand that goes in holding something and comes out empty flags the line.
• It never charges on a guess. Uncertain lines are marked "verify at exit". If the model is unavailable, the visit is marked "unchecked" rather than guessed.
• Exit check with an AI advisor. Nemotron-3-Super reads each flagged line's evidence and explains it in plain English, recommending keep, remove or ask the shopper. A person decides. Both go in the audit log, and the page shows how often the advisor and staff agree.
• The shopper's phone. A QR code opens the shopper's own live bill, with a "Not mine" button.
• Shelf audit. Between visits, Nemotron VL checks each shelf against the shop layout: it flags a product that doesn't belong there and reads the shelf price labels, flagging any that differ from what the bill charges. Staff resolve each finding.
• Shelf watch and layout. The same camera reads how full each shelf is and raises restock alerts. A floor plan keeps stock per shelf.
• Ask the shop. The owner asks "what's running low?" or "why were the takes on Aisle 2 uncertain?" A Nemotron agent answers with tool calls over the live shop data.

How I built it

The design idea is a funnel. A local pose model (YOLO11n-pose + ByteTrack) runs on every frame and finds hands. Nemotron is called only when a hand actually visits a shelf, so hosted inference costs $0.17 per 1,000 shelf visits. Decisions go through vjev, an open-source (Apache-2.0) typed-decision layer that I extended with a Token Factory backend. It reads Nemotron's answer logprobs, calibrates them into probabilities, abstains on timeouts, and records every call (model, latency, tokens, cost) for the console's model trace.

No GPU on the server. With the VLM hosted on Nebius Token Factory, pose tracking runs at 27.6 fps (320 px) on 4 vCPU. Aisle Agent runs on a $0.18/hour CPU instance plus Token Factory, and the demo server sleeps when idle and wakes on the first visit.

Results

Same items, hosted Nemotron vs a local 3B VLM on a T4:

Shelf audit, on built test sets: 48/48 pasted-in wrong-shelf products flagged with 0/48 false flags on clean shelves; price mismatches caught 26/26 (clean) and 24/26 (CCTV) when the price digits are 24 px tall, with no false flags from 9 px up. The full evaluation through Token Factory cost well under a dollar.

Challenges

• Reasoning on by default. With max_tokens=1 the first token was "The" and content was null. chat_template_kwargs={"enable_thinking": false} fixed yes/no answers, but free-text names still began "The user wants me to…"; asking for JSON fixed that.
• Made-up prices. Given the shelf's product list, the VL model tied labels to products much better but invented prices for labels too small to read. A label now counts only if a list-free reading saw the same price.
• Capacity. During one evaluation run, 68 of 380 calls failed with HTTP 529. Aisle Agent marks those visits "unchecked" instead of guessing.
• Take finding on close-up handheld footage is still the weak point. A fixed shop camera is the target.

What I learned

Calibrated logprobs from a hosted VLM are far more useful than free-text answers: they let the app abstain, flag "verify at exit", and never charge on a guess.

What's next

Real-shop footage, reading price labels from enlarged crops, per-shelf calibration, learning from staff decisions, and an edge box in the store that sends only the hard crops to Nemotron.

Built With

• amazon-ec2
• aws-lambda
• javascript
• nebius-token-factory
• nvidia-nemotron
• opencv
• python
• vjev
• yolo