Glass Pantry Vision Lab

Using OpenCV 5 to upgrade our fridge app: checks each shelf photo before the AI sees it, picks small labels for a closer look, and flags what may have changed. A person confirms every change.

Comment

Inspiration

Glass Pantry is a phone app we are building. It keeps a 3D twin of your fridge: each shelf, drawer and door bin, keeping track visually of what they contain. It is a work in progress. This project, the Glass Pantry Vision Lab, is where we test OpenCV 5 before it goes into the app.

The hard part of the app is keeping the twin true from photos. A vision model can list what is on a shelf, but it fails quietly. It guesses at labels too small to read and never doubts its own list. On our own photos a milk carton was read as soy sauce in one run and as eggs in the next. Reading the whole fridge again after each change costs time and money, and each re-read is another chance to be wrong.

So we put OpenCV 5 in the loop as a referee. It sends back a photo that cannot be read before any model call is spent, picks which small labels deserve a close-up, and compares a new photo with the last one to flag what may have changed. A person confirms everything, and nothing is deleted. We set each pass bar before opening the test photos and publish what failed. What passes here will go into the app.

What it does

The lab runs the app's workflow in a browser, with the vision engine on AWS. Five steps:

  1. Pick the fridge. Photograph the model sticker or the whole fridge. The AI reads the model and our catalogue supplies the layout (exact model, same family, or the closest generic layout).
  2. Trace each compartment. Drag a see-through outline onto the opening. OpenCV straightens the photo with no model call, and refuses an outline it cannot straighten. Then you line up the shelf lines yourself.
  3. Fill each shelf. Mark the shelf on the photo and only that strip is read. Before any model call, OpenCV's photo gate sends back a photo that is blurry, shaken, dark or full of glare. Each shelf line the model proposes is checked against real edges in the photo. Small labels get a close-up second look; blurred ones stay a question. You confirm the list before it is filed.
  4. Photograph it again: what changed? (experimental, still being improved) OpenCV 5's learned matching (ALIKED + LightGlue) lines the new photo up with the last one. No AI call is made. Each known thing gets a verdict (no visible change, possible change, moved, can't compare) and new spots are marked. Things that did not change keep their reading. A flag is only a question. If you mark a thing as gone it moves to an Unplaced list, never deleted, so you decide what happened.
  5. See it in 3D. The fridge is drawn with our app's own item models where the reading names one, and with a plain box or cylinder carrying its photo crop where it does not. After what changed?, the view moves to that shelf and plays the change.

How we built it

  • OpenCV 5 (opencv-contrib-python-headless 5.0.0.93). The photo gate is Laplacian and Sobel. The shelf check is Canny plus HoughLinesP (OpenCV 5's N×4 output). Straightening is getPerspectiveTransform and warpPerspective. Box size and sharpness pick the close-ups. For what changed?, the new top-level ALIKED and LightGlueMatcher classes run ONNX weights in OpenCV's DNN engine, then findHomography with USAC_MAGSAC and a per-item estimateAffinePartial2D fit. Classic SIFT and ORB, and ALIKED without LightGlue, are the baselines.
  • Agent loop. The model proposes. OpenCV's measurements pick the next step: retake, read a crop again, ask the person, or file. A person confirms each write.
  • AWS. One Lambda container image (4096 MB, CPU only) behind a Function URL. SAM deploys it from GitHub Actions through an OIDC role, with no stored keys. An owner switch in Parameter Store stops every live call. A shared daily cap in DynamoDB fails closed. CloudWatch logs hold codes and counts only, never photos or labels.
  • Models. Claude Opus 5.5 reads shelves. Claude Haiku 5.5 judges whether two label readings name the same product.
  • Browser. The fridge and the last confirmed photo of each shelf are kept in the browser's storage so you can come back. Every photo you upload is still sent to AWS, and to Anthropic when it is read. One tab writes, and a Forget button clears the stored copy. The 3D view is three.js with the app's 52 item models.

What we measured

We wrote down each pass bar before we opened the test photos. Then we ran 6 new before/after pairs.

  • OpenCV 5 lined up all 6, including one taken with the phone turned.
  • It found the removals (3 of 3) and the addition. It flagged the moved tub and marked its new spot. On the two pairs where nothing changed it raised no false item flags, though it marked three spots as new that were not.
  • Against OpenCV's classic SIFT matcher on the same photos, 15 needless questions against 50 (false item flags 8 vs 38, false new spots 7 vs 12).
  • 56 of 71 things kept their reading with no AI call. With SIFT, 27. With the code as shipped (one later rule sends a thing partly out of view to "can't compare"), 51 of 71 and 8 false new spots.
  • One OpenCV-only comparison of one pair took about 4 s on Lambda once warm (a spot check, not a distribution).

Later we compared against the alternatives: 12 pairings of 12 photos from one kitchen (the 6 pairs above and the same afternoon's photos re-paired, so a retrospective comparison on photos we had seen), protocol and answer key frozen first, three runs of each AI method, reviewed by Codex.

  • One Claude call given both photos and the old list was credited with every removal (5), move (2) and addition (5) in all three runs and asked about 27 to 29 of 148 unchanged items (2 to 3 change flags, the rest "can't compare"): about $0.03 and 4.7 s a pair.
  • OpenCV's first pass found the same removals and moves but asked about 66 unchanged items (31 change flags, 35 "can't compare"). The close-up reads that follow it cost about $0.09 and 16 s a pair (one pair went to a full read we did not measure). So that one call is planned as step two.
  • Among the model-free passes, the learned matcher cleared the most unchanged items under shared rules: 82 of 148. The same ALIKED keypoints with plain nearest-neighbour matching cleared 29 and could not line up 5 of 6 re-paired photos. SIFT and ORB, through rules set on the learned matches, cleared 33 and 20; ORB lined up all 12 pairs.
  • Re-reading the whole shelf and comparing names found the removals and additions but cannot see a move, and raised 24 to 36 false new spots per six pairs.
  • Limits: few photos, shared between pairings, and several physical changes scored more than once. Details in the report.

Challenges

False alarms are still above our bar, which is at most 1 false item flag and 2 false new spots per pair. They come from things a removed pack had covered, and from soft bags that shifted. So what changed? ships labelled experimental, and a flag is only a question. Two other ideas failed their bar on fresh test photos, automatic corner suggestions and shelf-edge snapping. We switched both off and let people place corners and lines by hand. In our first round a phone turned 90° between photos broke the learned matcher. We added a quarter-turn retry and re-tested on fresh pairs.

What we learned

OpenCV's job here is to referee, and that held. A blurry photo goes back before any model call. A shelf line with no edge under it becomes a question. Two photos that cannot be lined up are read in full instead of compared wrongly.

We also hoped OpenCV could clear things one by one so the AI is asked less. It could not: on the things the one-call method still asked about, OpenCV's pass cleared two item instances across the 12 pairs in each run, and every pair still needed questions or a full read. The other lesson: a check passes a bar we set before opening the test photos, or it stays off.

What's next

Bring the pieces that passed into the Glass Pantry app. The photo gate, sticker identify and outline editor go first, then what changed? with its second step as one AI call given both photos; whether the model-free first pass still saves a call in that flow gets measured. Test on more kitchens. Keep the twin up to date from rescans.

Team

We are 8yte: Juan Park in South Korea and Sadika in Bangladesh. Juan built the vision engine, the lab and the app; Sadika worked on the lab's front end and its readability.

Built With

Share this project:

Updates

Submission history