Smart Pantry

Inspiration

The inspiration for Smart Pantry stemmed from a universal frustration: not knowing which ingredients are currently in stock or which items need to be purchased as well as what can be done with current ingredients. Moreover, as engineering students, we understand that time is a valuable resource and that inventory is something that can easily be automated. So, we came up with an AI powered pantry to solve this problem.

What it does

Smart Pantry is an AI-powered inventory management system. When the pantry door is opened, a camera automatically captures an image of the shelf, which is processed by Gemini to identify inventory. This is stored and is used to recommend recipes and keep track of products that are going bad.

How we built it

The core of the system is a Rubik Pi C6490P running a Flask web server on Ubuntu. We integrated a MIPI CSI ribbon camera using a to handle image capture. For the "thinking" phase, we utilized Google Cloud Vertex AI and the gemini-2.5-flash-lite model for analysis. An ESP32 micro-controller relays door status and controls LEDs.

Challenges we ran into

Our main hurdle was the MIPI CSI camera producing unclear and unusable images, especially with changing light conditions. We solved this by including a 5-second period to stabilize exposure before capture. We also faced a "404 Client Error" when attempting to authenticate with Vertex AI using standard API keys; this required us to use the gcloud SDK.

Accomplishments that we're proud of

We are most proud of achieving a loop where a physical object, like a can of Diet Coke, is placed on a shelf and, within seconds, appears on a digital dashboard and is illuminated by a physical green light. Successfully connecting AI (Vertex AI) and low-level hardware (ESP32) wirelessly was a massive milestone for our team.

What we learned

We learned about the different roles computer hardware that operate at different abstraction levels. Having the Rubik Pi connect the ESP32 and locally host a site taught us about the use of POST to send commands to other devices on a local network.

What's next for Smart Pantry

In the future, Smart Pantry will be able to sync to a user's health statistics on their phones, this will allow for proper recommendations for meals based off exercise, whether the user wants to bulk or cut, calories burned, steps walked, etc. Moreover, this will be able to recommend groceries and create meal plans to help with diets, meal prepping, cooking for guests, etc. Also, this technology can be extended into refrigerators, shelves, spice cabinets, and other places where food is stored.