KitchenPal

"What should I eat?" — the simplest question, hardest to answer. KitchenPal combines vision, voice & chat AI to turn your fridge into personalized recipes.

Comment

KitchenPal: AI-Powered Recipe Generation & Cooking Assistant

Inspiration

"What should I eat?" — it's the simplest question, yet one of the hardest to answer. We've all stood in front of an open fridge, staring at random ingredients, wondering how they could possibly become a meal.

I wanted to solve this universal problem by creating an AI companion that could:

  • See what ingredients you have (via photo)
  • Understand your dietary needs and preferences
  • Suggest personalized recipes in natural conversation
  • Guide you through cooking with voice coaching

The Kiroween hackathon's Frankenstein category was the perfect fit — stitching together multiple AI technologies into one cohesive cooking assistant felt appropriately monstrous (in the best way).

What It Does

KitchenPal is a multi-modal AI cooking companion that transforms meal planning:

🍳 Chat with AI

Have natural conversations about what to cook. Tell it what ingredients you have, and get personalized recipe suggestions that respect your allergies and dietary restrictions.

📸 Snap Your Fridge

Take a photo of your ingredients or a dish you want to recreate. The Vision AI identifies ingredients and suggests recipes you can actually make.

🖼️ AI-Generated Food Images

Every recipe comes with a beautiful, appetizing image generated by Google's Imagen 3 — no more guessing what the final dish should look like.

🎙️ Voice Coaching

Hands covered in flour? No problem. ElevenLabs-powered voice coaching reads instructions step-by-step while you cook.

⚖️ Smart Portion Scaling

Adjust servings and watch all ingredient quantities recalculate in real-time.

🛒 One-Tap Grocery Ordering (Coming Soon)

Missing ingredients? Order them instantly through Instacart without leaving the app.

How We Built It

The Tech Stack

Layer Technology
Frontend Next.js 14, React 18, TypeScript, Tailwind CSS
AI - Text Google Gemini 1.5 Flash
AI - Vision Google Gemini 2.0 Flash
AI - Images Google Imagen 3
AI - Voice ElevenLabs Text-to-Speech
Backend Supabase (Auth + PostgreSQL + Storage)
Animations GSAP 3.13
Testing Vitest + fast-check (property-based testing)

Architecture

The system orchestrates $n = 4$ distinct AI services:

$$ \text{KitchenPal} = f(\text{Gemini}{\text{text}}, \text{Gemini}{\text{vision}}, \text{Imagen}_3, \text{ElevenLabs}) $$

┌─────────────────────────────────────────────────────────┐
│                   Next.js Frontend                      │
├──────────┬──────────┬──────────┬──────────┬────────────┤
│ Chat UI  │ Vision UI│ Recipe UI│ Voice UI │ Profile UI │
└────┬─────┴────┬─────┴────┬─────┴────┬─────┴─────┬──────┘
     │          │          │          │           │
┌────▼─────┬────▼─────┬────▼─────┬────▼─────┬─────▼──────┐
│ /api/chat│/api/vision│/api/image│/api/voice│  Supabase  │
└────┬─────┴────┬─────┴────┬─────┴────┬─────┴─────┬──────┘
     │          │          │          │           │
┌────▼─────┬────▼─────┬────▼─────┬────▼─────┬─────▼──────┐
│  Gemini  │  Gemini  │  Imagen  │ ElevenLabs│ PostgreSQL │
│ 1.5 Flash│ 2.0 Flash│    3     │    TTS    │  + Auth    │
└──────────┴──────────┴──────────┴──────────┴────────────┘

Kiro-Powered Development

I leveraged Kiro's features extensively:

Spec-Driven Development (/.kiro/specs/)

  • Created detailed requirements with user stories and acceptance criteria
  • Designed technical architecture with TypeScript interfaces
  • Defined $25+$ correctness properties for property-based testing

Steering Documents (/.kiro/steering/)

  • color-palette.md — Material Design 3-inspired color system
  • animations.md — GSAP animation standards and pre-built components
  • auth-flow.md — Authentication patterns and session management

Challenges We Ran Into

1. Multi-AI Orchestration

Coordinating four different AI services with different response times, rate limits, and error modes was complex. The total latency is bounded by:

$$ T_{\text{total}} \leq \max(T_{\text{Gemini}}, T_{\text{Imagen}}, T_{\text{ElevenLabs}}) + T_{\text{overhead}} $$

I implemented:

  • Exponential backoff for rate limiting: $t_{\text{wait}} = \min(t_{\text{base}} \cdot 2^n, t_{\text{max}})$
  • Response caching for common queries
  • Graceful fallbacks when services fail

2. Image Persistence

AI-generated images from Imagen 3 come as base64 strings. Displaying them directly caused CORS issues and performance problems.

Solution: Automatically persist generated images to Supabase Storage and serve them via URL.

3. Context-Aware Conversations

Making the AI "remember" user preferences (allergies, dietary restrictions) across conversations required careful prompt engineering. I built a PromptBuilder service that injects user context into every request:

$$ \text{Prompt}_{\text{final}} = \text{SystemPrompt} \oplus \text{UserContext} \oplus \text{Query} $$

Where $\oplus$ denotes prompt concatenation with appropriate delimiters.

4. Real-Time Portion Scaling

Ingredient quantities needed to scale proportionally when users adjust portions:

$$ Q_{\text{new}} = Q_{\text{original}} \times \frac{S_{\text{desired}}}{S_{\text{original}}} $$

Where:

  • $Q$ = ingredient quantity
  • $S$ = number of servings

This required parsing AI-generated ingredient data into structured formats and implementing live recalculation.

Accomplishments That We're Proud Of

🏆 True Multi-Modal AI Integration

Successfully stitched together five different services (Gemini text, Gemini vision, Imagen 3, ElevenLabs, Supabase) into a seamless user experience. Each AI handles what it does best.

🏆 Production-Ready Architecture

This isn't a hackathon prototype — it's built with:

  • Rate limiting with exponential backoff
  • Response caching for performance
  • Row-Level Security (RLS) policies
  • Comprehensive error handling
  • Full TypeScript type safety

🏆 Spec-Driven Development with Kiro

Created detailed specifications before writing code. The /.kiro directory contains:

  • $2$ complete feature specs with $18$ user stories
  • $25+$ correctness properties for testing
  • $3$ steering documents for consistent code generation

🏆 Property-Based Testing

Used fast-check library to verify correctness properties — catching edge cases that traditional unit tests miss.

🏆 Delightful UX Details

  • GSAP-powered buttery animations
  • Voice coaching for hands-free cooking
  • Live portion scaling with smooth transitions
  • Material Design 3-inspired color system

What We Learned

  1. Spec-first development pays off — Writing detailed specs before coding made implementation predictable and reduced rework

  2. Steering documents ensure consistency — Kiro-generated code followed design standards without manual correction

  3. Property-based testing catches edge cases — Using fast-check to verify correctness properties found bugs that unit tests missed

  4. Multi-modal AI is powerful but complex — Combining different AI modalities creates magical experiences, but requires careful error handling

  5. Kiro accelerates development — The combination of specs, steering, and vibe coding made building a complex multi-AI app manageable

What's Next for KitchenPal

  • Instacart Integration — One-tap ordering of missing ingredients directly from your recipe
  • Meal planning calendar — Plan your week's meals with automatic grocery lists
  • Nutritional tracking — Integration with health apps
  • Community recipes — Share and discover recipes from other users
  • Smart kitchen integration — Connect with smart appliances for guided cooking
  • Offline mode — Cache recipes for cooking without internet
  • Multi-language support — Voice coaching in multiple languages

Built With

Share this project:

Updates