FridgeSense
Every door opening, with proof
Ask your fridge
Teach camera your fridge
Fridge habits at a glance
Fridge TV dashboard
How it works
Where to put the camera

Inspiration

We've all stood in a grocery aisle wondering "do we still have soda?", or come home to find someone finished the last can and didn't say anything. Smart fridges with built-in cameras cost thousands of dollars and only show you a blurry shelf photo. Plenty of homes already have a small indoor camera, though. We wanted to see whether one camera pointed at a normal fridge could keep an accurate inventory on its own, with no barcodes, no typing and no new appliance.

What it does

FridgeSense turns a Ring Indoor Cam 2K pointed at the fridge into a fridge that remembers.

Every door opening is analyzed. When the door opens, FridgeSense gets the clip from Ring, confirms the door really opened, and asks Amazon Nova 2 Lite what went in or came out.
Running inventory on your phone. A PWA shows what's in stock by category, what ran out, and an Activity feed with "proof" stills of each change ("+ soda", "− 2 eggs").
Ask by voice. A self-hosted MCP server (spec 2025-11-25, Streamable HTTP) lets Alexa+ or any MCP client answer "Do I have soda?", "What's running low?" or "When was the fridge last opened?". The app's Ask screen does the same with Nova as the MCP client and Amazon Polly reading the answer aloud.
Fridge stats. Opens per day, how long the door stays open, and a day-by-hour heatmap.
Kitchen TV dashboard. A read-only Fire TV view of stock, recent changes and today's opens, readable from across the room.
Review instead of guessing. Changes Nova isn't confident about wait on a Review screen for one-tap confirmation.

How we built it

Everything is serverless in AWS us-east-1 and deployed with one cdk deploy.

Ring webhook → API Gateway + Lambda. We verify the X-Signature HMAC first, skip duplicates by request_id, reply 200 right away and start one Step Functions run.
Door check. The run downloads the Ring clip once and checks door zones every 0.5 s with a pixel rule you set up in the app. Motion without a door opening stops here, so walking past the fridge costs nothing.
Privacy first. Up to 4 stills from the open stretch are cropped to the fridge and face-blurred with Amazon Rekognition before anything is stored. Clips are blurred by a container Lambda running an open-source face detector. Stills and clips are deleted from S3 after 7 days, and live video is owner-only.
Nova 2 Lite watches a short, 2× slow-motion video of the open stretch, guided by reference photos of items in this fridge. It describes what the hands did, then lists each item seen going in or coming out. Only one-way hand evidence changes the inventory, so items just sitting on a shelf never get counted.
DynamoDB (single table keyed by Ring account and device) feeds the PWA (React, Vite, Tailwind, Recharts on S3 + CloudFront) and the MCP server, so the app, voice and TV always give the same answer.

A typical door opening costs about a fifth of a cent: about $0.0014 for Nova, $0.001 per still for face detection, and a fraction of a cent to blur the clip. Every paid call logs its own cost.

We built it spec-first with OpenSpec and test-first throughout: pytest, moto and Vitest, using recorded webhooks and canned Bedrock responses, so tests never touch live Ring or AWS.

Challenges we ran into

"Motion" isn't "door opened." People walk past the fridge all day. A door-zone check on clip frames, set up and calibrated from the app, filters out most motion before any paid call is made.
Which way did the soda go? Early versions compared before and after shelf photos and got confused by things being moved around. Switching to hand evidence (what was seen in a hand, and which way it was moving) made the direction reliable.
Ring API surprises. The token exchange request format, clip downloads that answer 303 or 206, stills that don't line up with clip frames, and no way to get a still of the camera "now". We worked around each one and logged them all.
Bedrock details. Nova 2 Lite has no structured outputs, so we used Converse tool use with Pydantic validation. It needs the cross-region inference profile in us-east-1, and prompt caching has rules the docs don't mention.
MCP on Lambda. MCP Python SDK v2 broke v1 code, and Strands couldn't share a project with mcp 2.3, so the Ask agent uses Bedrock Converse tool use directly.
Alexa+ access. The MCP Toolkit setup looked gated for hackathon entrants, so we also built the in-app Ask experience on the same MCP server.

We recorded 33 friction-log entries with exact steps and suggestions, and submitted them with this entry.

What we learned

A cheap deterministic check in front of a multimodal model saves most of the cost and most of the mistakes.
Asking a vision model to describe evidence first ("what did the hands do?") and decide second works much better than asking for the answer directly.
MCP is a great seam: one small read-only server gave us voice, a chat screen and desktop clients without changing anything else.

What's next

Multi-fridge and multi-camera households, expiry tracking ("what should I eat first?" already works from item age), shopping-list export, and a public Ring Appstore app with proper account linking.