Smart Fridge Alexa

Inspiration

Households throw away over 1 billion meals every day (UNEP Food Waste Index 2024), and a lot of that is food that spoiled in the fridge. Adults also spend close to 40 hours a year just deciding what to eat (UK survey). We wanted a voice assistant that knows what is in the fridge and what is about to expire, without anyone typing in an inventory.

What it does

A camera image of the fridge is classified by a vision model. The result becomes a live inventory that an Alexa+ style assistant can query by voice: what is in the fridge, what is about to expire, whether we have a specific item, and what to cook for dinner. Recipe suggestions put items that are about to spoil first, list what you have and what is missing, and give step-by-step instructions.

How we built it

The project has four parts, connected through the Model Context Protocol (MCP).

The Alexa+ simulator is a web UI plus an LLM agent that acts as an MCP client. It discovers the server's tools and calls them, and it shows a trace of every tool call with timing, flagging any call slower than 500 ms.

The MCP server is a Streamable HTTP server exposing 11 tools for the fridge (inventory, check an item, expiring items, corrections, system status), dishes (suggest recipes, recipe steps, online recipe search) and a shopping list. It caches the camera data and refreshes it in the background so answers stay fast.

The dish agent takes the inventory with days left and the user's preferences and returns up to three dishes with reasons, what you have, what is missing, and steps. With an LLM it is an agent with four tools; without one it falls back to ranking 14 built-in recipes by how much they use expiring items.

The image classifier and Edge API send fridge images to a Claude vision model by default (Sonnet 5.5), with optional local detectors. The API returns normalized labels with confidence scores, for example folding "red onion" into "onion", and it tracks token cost per call.

Built to keep working

Every LLM-backed part uses a provider chain: Amazon Bedrock through the Strands Agents SDK if AWS credentials exist, then the Anthropic API, then OpenAI, then a built-in keyword router and recipe rules. A provider that fails with an access or quota error is skipped for five minutes, so one broken service does not slow every answer. If the camera is unreachable, the assistant says so instead of claiming the fridge is empty. The repo has over 100 automated tests that run offline and do not read a developer's real .env file.

Amazon Bedrock and Strands

The dish agent and the simulator both have a Strands Agents path on Bedrock (Claude Sonnet 4.6 for dishes, Claude Haiku 4.5 for the simulator), tested with a scripted model against the real MCP server. Our own AWS accounts could not call Bedrock models before the deadline (details in the friction log), so the demo runs on the Anthropic API. Anyone with Bedrock access can switch by setting the provider to auto and adding AWS credentials.

Challenges

Keeping answers fast when the camera or an LLM is slow, handling inconsistent labels from vision models, and keeping the whole system working when an API key or provider is unavailable.

What's next

Multiple images per fridge compartment with a higher-confidence merge, preferences that persist between sessions, a larger recipe set, and a real Alexa+ skill in place of the simulator.

Built With

• amazon-web-services
• anthropic
• bedrock
• claude
• fastapi
• mcp
• python
• strands