Mr. Fridge

• 

  Detection example

Inspiration

I started living alone recently, and my fridge turned into a small disaster. I'd buy vegetables, forget them behind something, and find them a week later already gone bad. Or I'd be halfway through cooking and realize I was out of eggs. I wanted the fridge to just keep track of itself: what's in it, roughly how old things are, what's running low, all without me scanning barcodes or typing into an app. That's Mr. Fridge.

What it does

Mr. Fridge watches the inside of your fridge and keeps a running inventory of what's there.

It's event-driven, so nothing happens until the door opens. An ESP32-S3 camera node sits on the ceiling of the fridge, asleep with its camera and Wi-Fi off, and a magnetic reed switch on the door wakes it. Open the door and it records the moment from above and uploads the frames; close it and that triggers the final scan and updates the inventory. Don't want to install hardware? Just prop a phone inside the fridge instead. Same flow, and it's what we use for demos.

The hub runs those frames through a vision model to get labels and boxes, then reconciles the inventory: what got added, what got removed, and roughly how old each thing is (it compares when an item first showed up against a shelf-life table). It only estimates from what it actually saw, and you can correct it when it's unsure.

There's also an optional planner that looks at your inventory, preferences, and budget and proposes a restock cart. The key part is that the model only proposes. The rules that actually matter (allergens, dislikes, spending limits, duplicates, approvals) live in plain application code, not in the model.

How we built it

The hub is a Python service on FastAPI, running on Railway in Docker, with SQLite holding the inventory and the event frames. The dashboard is deliberately dependency-free (plain JavaScript, HTML, and CSS), so it loads fast on a phone wedged in a fridge.

For recognition we use NVIDIA's LocateAnything-3B on a separate GPU service, hosted on RunPod since the model wants a 48GB card, and the hub talks to it over an authenticated HTTP endpoint. When there's no GPU around, like in the lightweight hosted demo, it falls back to a vision model through OpenRouter, and the dashboard always shows which backend is actually live so we never pass the fallback off as LocateAnything.

The optional restock planner runs on GPT-5.6 (openai/gpt-5.6-luna) through OpenRouter and returns a strict, typed plan that our own code checks against the allergen, budget, and approval rules before anything can happen. Retail is deliberately fenced in: catalog lookups, an Instacart handoff, and Stripe in test mode. No unattended real purchases.

The camera node is an ESP32S3 CAM V1.0, flashed with PlatformIO. It deep-sleeps on the door contact, wakes on open, asks for 1080p when the sensor supports it, and streams JPEGs to the hub.

The whole thing was built in Codex on GPT-5.6: the firmware, the phone light-trigger flow, the inventory reconciliation, the dashboards, the vision integration, the tests, and the deploy troubleshooting. We set the plan and the bar each step had to clear; Codex made scoped changes and ran the checks. The safety-critical logic stayed in ordinary, reviewable code on purpose.

Challenges we ran into

Getting the vision to stop double-counting took real work. A stack of cartons would read as extra items, so we lean on the model for "what and where" and keep the counting logic on our side.

The ESP32-S3 was its own adventure. It's a camera-only board with no built-in door sensor, so the whole design leans on an external reed switch and deep sleep to survive on a battery.

But the biggest lesson was about honesty. One camera on the ceiling can't see behind a package or through a shelf, and there's no gas or smell sensor, so it genuinely cannot tell you food is safe, only what it can see. We wrote all of that into the docs instead of hiding it, and kept age estimates separate from any food-safety claim.

Accomplishments that we're proud of

A real end-to-end path: the door opens, frames go up, a real vision model returns boxes, and the inventory updates, with a phone fallback so anyone can try it without the hardware.

Keeping the model on a short leash. It proposes objects and restock ideas; our code decides. Allergens, budgets, and checkout stay deterministic and reviewable.

What we learned

Keep the risky decisions in code you can test, not in the model. It's great at "what's in this photo" and bad at "is this safe to eat," so we only ask it the first kind of question.

And write down what doesn't work. Being honest about what the camera can't see made the whole thing more trustworthy, not less.

What's next for Mr. Fridge

Getting the LocateAnything GPU path running as the default instead of the fallback, finishing the battery-powered ESP32-S3 node so it doesn't need to stay plugged in, and turning the guarded test-mode restock flow into something that can actually keep the fridge stocked.

Built With

• codex
• css
• esp32
• gemini
• gpt-5.6
• html
• javascript
• openai
• openrouter
• python
• railway