PantryPal - Devpost Project Story

💡 Inspiration

Every year, the average household throws away approximately $1,500 worth of food. That's not just money going in the trash—it's resources, labor, and environmental impact wasted. We've all experienced it: buying groceries, forgetting what's already in the fridge, and discovering expired items weeks later.

We asked ourselves: What if your kitchen could be smarter?

The idea for PantryPal was born from a simple frustration—standing in a grocery store, trying to remember what we already had at home, and coming back with duplicates while the items we actually needed sat forgotten. We wanted to build something that would:

Eliminate the mental load of tracking what's in your kitchen
Reduce food waste by surfacing items before they expire
Inspire creativity by suggesting recipes based on what you actually have
Save time by automating the tedious parts of meal planning

🍳 What It Does

PantryPal is your AI-powered smart kitchen companion that transforms how you manage food and cook meals.

Core Features

Feature	Description
📷 Receipt Scanning	Snap a photo of your grocery receipt and watch as AI extracts every item automatically
📦 Smart Pantry	Track everything in your kitchen with quantities, categories, and expiry dates
🤖 AI Recipe Suggestions	Get 3 personalized recipes based on exactly what's in your pantry
📥 Recipe Import	Paste any recipe URL or YouTube cooking video—we extract the full recipe
👨‍🍳 Cooking Mode	Step-by-step guided cooking with progress tracking
🛒 Shopping List	Automatically add missing ingredients with one click
⚠️ Expiry Alerts	Visual warnings for items expiring within 3 days

The Magic Formula

$$ \text{Food Waste Reduced} = \text{Smart Tracking} + \text{AI Suggestions} + \text{Timely Reminders} $$

🛠️ How We Built It

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        Frontend                              │
│  React 19 + Vite + Tailwind CSS + Firebase Auth             │
└─────────────────────┬───────────────────────────────────────┘
                      │ REST API
┌─────────────────────▼───────────────────────────────────────┐
│                    Flask Backend                             │
│  Python + Flask-CORS + Request Handling                     │
└─────────────────────┬───────────────────────────────────────┘
                      │ API Calls
┌─────────────────────▼───────────────────────────────────────┐
│                   Gumloop AI Pipelines                       │
│  Receipt OCR │ Recipe Extraction │ Recipe Suggestions       │
└─────────────────────────────────────────────────────────────┘

Tech Stack

Frontend:

React 19 with modern hooks and functional components
Vite 7 for blazing-fast development
Tailwind CSS 4 for beautiful, responsive design
Firebase for authentication (Google Sign-In) and Firestore database
Lucide React for consistent, beautiful icons
Recharts for data visualization

Backend:

Flask as our lightweight Python API server
Gumloop for orchestrating AI pipelines
Custom pipelines for OCR, recipe extraction, and AI suggestions

AI Pipelines:

Receipt OCR Pipeline
- Accepts receipt images
- Extracts structured data: food_name, quantity, unit, food_category
- Returns CSV format for easy parsing
Recipe Extraction Pipeline
- Accepts any URL (websites, YouTube videos)
- Uses AI to parse and structure recipe data
- Returns JSON with ingredients, instructions, timing
Recipe Suggestion Pipeline
- Takes pantry contents as CSV input
- AI generates 3 complete recipes using available ingredients
- Returns full recipes with links to original sources

Data Model

// Pantry Item Schema
{
  name: "Chicken Breast",
  quantity: 2,
  unit: "lbs",
  category: "protein",  // protein | dairy | grain | fruit | vegetable | other
  expiryDate: "2026-01-25",
  createdAt: Timestamp
}

// Recipe Schema
{
  recipe_title: "Garlic Butter Chicken",
  description: "A quick and flavorful weeknight dinner",
  prep_time_minutes: 15,
  cook_time_minutes: 25,
  servings: 4,
  ingredients: [
    { name: "chicken breast", quantity: 2, unit: "lbs", preparation_notes: "cubed" }
  ],
  instructions: [
    { step_number: 1, instruction_text: "Season chicken with salt and pepper" }
  ]
}

🚧 Challenges We Faced

1. Receipt OCR Accuracy

Grocery receipts are messy. Faded ink, crumpled paper, varying formats across stores—our initial OCR attempts returned gibberish half the time.

Solution: We leveraged Gumloop's AI pipeline capabilities to build a robust OCR system that not only reads text but understands context. The AI knows that "BNLS CHKN BRST" means "Boneless Chicken Breast" and categorizes it correctly.

2. Recipe Extraction from Diverse Sources

Recipe websites are infamous for burying the actual recipe under walls of text, ads, and life stories. YouTube videos add another layer of complexity—no structured data at all!

Solution: Our Gumloop pipeline uses AI to intelligently parse any source:

For websites: Extracts structured recipe data regardless of page layout
For YouTube: Analyzes video descriptions and transcripts to reconstruct the full recipe

3. Real-time State Management

Keeping the pantry synchronized across components while handling Firebase's asynchronous nature was tricky.

Solution: We implemented a clean architecture with:

Context providers for global auth state
Firebase service modules for clean data access patterns
Optimistic UI updates for snappy user experience

4. Cross-Platform Compatibility

We needed the app to run seamlessly on Mac, Windows, and Linux with various Python environments.

Solution: We built a Python-based cross-platform launcher (start.py) that:

Auto-detects virtual environments
Finds the correct Python interpreter
Manages both frontend and backend processes
Handles graceful shutdown on Ctrl+C

🏆 Accomplishments We're Proud Of

End-to-end AI Integration: Three working AI pipelines that transform images and URLs into structured data
Beautiful, Responsive UI: A polished dark-mode interface that feels native and modern
Full Firebase Integration: Real-time data sync with proper authentication
Cooking Mode: Step-by-step guided cooking experience with progress tracking
One-Click Everything: Add items from receipt, add missing ingredients to shopping list, save recipes to library—all with single clicks

📚 What We Learned

AI Pipeline Design: Building robust AI workflows that handle edge cases gracefully
Gumloop Integration: Orchestrating multiple AI services through a unified pipeline system
Modern React Patterns: Leveraging React 19's latest features with hooks and contexts
Firebase Architecture: Structuring Firestore data for optimal query performance
Cross-Platform Development: Building tools that work seamlessly across operating systems
UX for Complex Features: Making AI-powered features feel simple and intuitive

🔮 What's Next for PantryPal

Feature	Description
📊 Nutrition Tracking	AI-powered macro and calorie tracking based on pantry contents
🤝 Family Sharing	Share pantry access with household members
📅 Meal Planning	Weekly meal planning with automatic shopping list generation
🏪 Store Integration	Connect with grocery stores for price comparisons and ordering
🗣️ Voice Commands	"Hey PantryPal, what can I make for dinner tonight?"
📱 Mobile App	Native iOS and Android apps with barcode scanning
🌍 Food Donation	Connect expiring items with local food banks