Pantry Twin

Inspiration

We wanted to make grocery shopping and pantry management smarter and easier. Many people buy groceries and forget what they already have or when items will expire. Pantry Twin was inspired by that everyday struggle — to reduce waste, manage expiration dates, and give intelligent grocery suggestions using AI.

What it does?

Pantry Twin scans grocery receipts and automatically updates your virtual pantry. It tracks expiry dates, identifies soon-to-expire items, and builds a “Smart Grocery List.” Users can also ask questions through an AI chatbot, like “Do I still have milk?” or “Add eggs to my grocery list,” and the app handles it instantly.

How we built it.

We built the backend with Flask in Python, handling endpoints for pantry data, grocery lists, and AI-based conversations. The Gemini API extracts text from receipt images and understands natural queries using structured tool calls. The frontend was developed with React, communicating with Flask through REST APIs. We used Flask-CORS for cross-origin requests and mock data for testing inventory behaviour.

Challenges we ran into

Integrating Gemini Vision OCR was one of the hardest parts — ensuring accurate JSON parsing and filtering out subtotal lines was tricky. We also had to manage file uploads, clean-up, and handle unexpected response formats. Another challenge was connecting conversational tool functions (like checking inventory or adding grocery items) with Gemini’s structured outputs while keeping it responsive in real time.

Accomplishments that we're proud of

We built a working system that scans real receipts, detects food items, and interacts naturally like a smart kitchen assistant. The Flask–Gemini integration worked smoothly, and the frontend displays data in real time. The chatbot understanding real human-like queries was a big milestone.

What we learned

We learned how to combine AI multimodal APIs with real-world applications, handle JSON schemas, manage backend–frontend communication securely, and deploy machine learning-based features into everyday utilities. We also explored how LLM tools can make traditional apps more dynamic.

What's next for Pantry Twin

Next, we plan to add barcode scanning, multi-user cloud sync, nutrition suggestions, and personalized recipes based on what’s already in your pantry. The long-term goal is to turn Pantry Twin into a full home grocery manager powered by vision and conversation AI.