CookWithMe

CookWithMe is a multi-agent, multimodal shopping assistant that parses requests, expands recipes, reads UIs, and builds carts automatically on Blinkit & Zepto.

Comment

🍳 CookWithMe: Multi-Agent Multimodal Grocery Assistant

💡 Inspiration

Anyone who’s tried cooking after a long day knows the drill: find a recipe, open Blinkit or Zepto, search each ingredient one by one, compare pack sizes and prices, and finally checkout — only to find some items out of stock.

The friction isn’t logistics — it’s cognitive load. CookWithMe asks a simple question: what if you could just say "make pav bhaji for 4" and come back to a full cart?

India’s quick-commerce ecosystem (10–30 min delivery, rich SKU catalogues) is the perfect playground. Gemini’s multimodal reasoning — understanding screenshots, UI context, and clicks — makes this agent possible.

🛒 What It Does

CookWithMe is a conversational, multimodal shopping agent. You talk in plain English (or Hinglish), and it shops for you.

Typical session:

  1. You type: "Make butter chicken for 4 people"
  2. CookWithMe parses intent, expands the recipe into an ingredient list, and asks for confirmation.
  3. You select platform: Blinkit, Zepto, or both.
  4. The agent shops: searches, reads screenshots, selects best products & pack sizes, clicks Add, verifies success, and moves to next item.
  5. Self-correction handles unavailable items, wrong pages, or missing products.
  6. You get a live cart summary: items, quantities, prices, fees, and serviceability status.

Key capabilities:

  • Natural language → structured shopping list
  • Recipe expansion with per-serving quantity math
  • Multimodal Gemini vision for product selection & UI interaction
  • Pack-size reasoning for exact quantities
  • Substitution for unavailable items
  • Cross-platform comparison with full fee breakdown
  • Live progress streaming via SSE
  • Session persistence & personalization

🏗 How I Built It

Architecture: Multi-Agent, Vision-First

Agent Responsibility
IntentParserAgent Free-text → structured shopping intent
RecipeExpanderAgent Recipe + servings → full ingredient list
FusedVisionAgent Screenshots → page state, product selection, ADD button, quantity plan
SubstitutionAgent Finds best recipe-compatible alternatives
CoreLoop Orchestrates search → analyse → click → verify → self-correct

Vision Loop (per item)

search results screenshot
        │
        ▼
FusedVisionAgent.analyse()
  → page_state
  → selected_product + pack_size
  → combination_plan (units needed)
  → add_button_x_norm, add_button_y_norm
        │
        ▼
BrowserManager.click_add_button()   ← DOM-first, vision coords as fallback
        │
        ▼
FusedVisionAgent.verify_add()        ← before/after screenshot comparison
  → success? stepper appeared? cart badge incremented?
        │
   if failed:
        ▼
_self_correct_add()
  → go_back if navigated to product page
  → re-analyse to get fresh coordinates
  → retry with DOM click

🛠 Technology Stack

  • Gemini 3.1 Flash Lite — vision & text reasoning
  • Google Gen AI Python SDK — async wrapper, rate limiting
  • Playwright — headless Chrome automation
  • FastAPI + uvicorn — WebSocket + SSE dashboard
  • Pydantic v2 — strict typed models
  • structlog — structured logs for all agent decisions
  • Docker + noVNC — containerised browser for demo & Google Cloud for deployment

⚡ Challenges

  1. 🌐 Diverse Platform UIs — CSS, DOM, and patterns differ. Three-tier strategy: CSS → text scan → visual fallback.
  2. 🖥 Dynamic SPAs — inconsistent ADD confirmations; multi-signal verification with Gemini vision + DOM.
  3. 🛍 Cart Navigation — Zepto product card navigates away; vision model localizes ADD button + recovery path.
  4. 🔢 Quantity Arithmetic — Recipes rarely match stock sizes. A combination planner lets the agent pick the most efficient pack mix. LLM chooses the product; Python computes exact units; cross-checked before adding.
  5. 💬 Conversational Context — tracks pre-existing cart, session additions, and new requests.
  6. 📡 Real-Time Streaming — SSE + WebSockets maintain live dashboard across async multi-agent pipeline.

🏆 Accomplishments

  • Genuinely agentic loop: sees, reasons, and decides from screenshots.
  • One Gemini model, five agents: text parsing, recipe expansion, vision, verification, substitution.
  • Robust quantity math: handles partial packs efficiently.
  • Self-correction: retries, navigates back, re-analyzes.
  • Cross-platform comparison: Blinkit & Zepto with full fees, coverage, and recommendation.
  • Live dashboard: chat + mirrored browser shows agent progress in real-time.

📚 What I Learned

  • Multimodal grounding changes the problem — vision removes hallucinations but adds UI-specific failure modes.
  • Structured output is essential — Pydantic schemas enforce correctness, preventing silent downstream errors.
  • Google Cloud deployment & ecosystem — got hands-on experience with Playwright automation, Cloud Run deployment, and Gen AI Python SDK, which I wasn’t familiar with before.

🚀 What's Next

  • Voice input — hands-free cooking commands
  • Meal planning mode — optimised weekly carts
  • Price memory & trend alerts — notify discounts per SKU
  • Agent-as-API — REST/webhook for third-party apps
  • Multi-user session isolation — cloud-managed concurrent browsing

Built With

Share this project:

Updates

Submission history