-
-
finance
-
kitchen chef
-
inventory
-
dark theme
-
dashboard
-
voice agent
-
finance agent
-
ai assist in inventory
-
ai image creation agent agent after recipt scan it gen image and check the inventory for interval if there is aimage in db it would pull tr
-
customizer
Inspiration
Every Indian household carries a silent tax: the mental load of knowing what's in the fridge, what's about to expire, what to cook tonight, and how much was spent at the kirana store last week. My mother keeps a physical notebook for this. My flatmates argue every Sunday about what groceries to buy. I've thrown away spinach I forgot I bought three days ago.The problem isn't laziness. It's that household intelligence is distributed across a dozen disconnected places — receipts in a drawer, expiry dates in your head, nutrition goals in a fitness app, budgets in a spreadsheet. I wanted to build a system that held all of this together and reasoned across it, the way a genuinely helpful household manager would.PantryMind is that system.
What it does
PantryMind is a multi-agent AI life management system for Indian households, built as a local-first Electron desktop app. It manages:
Pantry inventory with expiry tracking calibrated for Indian grocery categories (dal, paneer, methi, mutton — not just "produce" and "dairy") AI receipt scanning via Gemini Vision that extracts items from thermal-paper Indian receipts with GST/SGST line detection Meal planning powered by a linear programming optimizer (PuLP) that maximizes protein within calorie and macro constraints using exactly what you have in the fridge Finance tracking that automatically logs every grocery run, categorizes spending in INR, and answers natural language questions like "how much did I spend on meat this month?" A real-time voice agent backed by Gemini Live Bidi streaming, so you can ask "what can I cook tonight?" hands-free while cooking
The core architecture is a Google ADK orchestrator that routes every user message to a specialist sub-agent: pantry_agent, kitchen_chef_agent, finance_agent, receipt_agent, shopping_list_agent. All agents share a governed MongoDB MCP toolset that enforces a human-in-the-loop approval layer for any write operation above a policy threshold.
How we built it
The backbone is a FastAPI application running five ADK LlmAgent instances, all backed by gemini-2.5-flash on Vertex AI. Agents communicate with MongoDB Atlas via the @mongodb-js/mcp-server-mongodb MCP server, launched as a stdio child process. Every write tool (insertOne, insertMany, updateOne) is wrapped in a governance layer that:
Sanitizes the document Evaluates action policy — risk level, INR value threshold, destructive flag Either auto-approves or creates a pending_action that surfaces in a frontend Approval Inbox for the user to confirm
The meal optimizer is a separate section — paste it after, with a clear header:
Meal Optimizer: Linear Programming with PuLP Most "AI meal planners" just prompt an LLM. PantryMind uses a real LP solver. Decision variables ( x_i ) represent grams of ingredient ( i ), paired with binary variables ( u_i \in {0,1} ) indicating whether an ingredient is included at all. Objective — maximise protein: maximize∑ixi⋅pi\text{maximize} \quad \sum_{i} x_i \cdot p_imaximizei∑xi⋅pi Calorie target (±10% band): 0.9⋅Ctarget≤∑ixi⋅ci≤1.1⋅Ctarget0.9 \cdot C_{\text{target}} \leq \sum_{i} x_i \cdot c_i \leq 1.1 \cdot C_{\text{target}}0.9⋅Ctarget≤i∑xi⋅ci≤1.1⋅Ctarget Macro floor and ceilings: ∑ixi⋅proteini≥Pmin\sum_{i} x_i \cdot \text{protein}i \geq P{\min}i∑xi⋅proteini≥Pmin ∑ixi⋅carbsi≤Carbsmax\sum_{i} x_i \cdot \text{carbs}i \leq \text{Carbs}{\max}i∑xi⋅carbsi≤Carbsmax ∑ixi⋅fati≤Fatmax\sum_{i} x_i \cdot \text{fat}i \leq \text{Fat}{\max}i∑xi⋅fati≤Fatmax Big-M linking — ingredient is either fully in or fully out: xi≥ui⋅mmin,xi≤ui⋅M∀ ix_i \geq u_i \cdot m_{\min}, \qquad x_i \leq u_i \cdot M \quad \forall\, ixi≥ui⋅mmin,xi≤ui⋅M∀i Variety and dominance constraints: ∑iui≤Kmax\sum_{i} u_i \leq K_{\max}i∑ui≤Kmax ∑i∈protein sourcesui≤2\sum_{i \in \text{protein sources}} u_i \leq 2i∈protein sources∑ui≤2
Sanitizes the document Evaluates action policy (risk level, INR value threshold, destructive flag) Either auto-approves or creates a pending_action that lands in a frontend Approval Inbox
For semantic ingredient search, I built an Atlas Vector Search pipeline over the inventory collection using 768-dimensional text-embedding-004 embeddings. This lets the Kitchen Agent answer "what do I have for a Thai curry?" without needing a structured query — it just finds semantically similar inventory items by cosine similarity. The meal optimizer is a real LP solver:
Challenges we ran into
Indian receipt OCR is genuinely hard. Indian thermal receipts are chaotic. GST and SGST line items (e.g., CGST 2.5% on ₹340) get misread as product names. Store names print in ALL CAPS with no spaces. Quantities appear as 500GMS or 1NOS or 1KG. I had to write an explicit multi-step RECEIPT_OCR_PROMPT that instructs Gemini to skip lines matching (CGST|SGST|IGST|TAX|DISC|MRP|TOTAL|AMOUNT|GST|%) and parse only lines with item names, quantities, and unit prices. I also preprocessed images client-side with contrast/brightness filters tuned for thermal paper before sending to the API.
Voice model ASR failures break tool calls. The Gemini Live model frequently garbles low-energy speech ("make me a meal" → "Mir ja" in the transcript), causing it to miss the delegate_to_pantrymind tool call entirely and respond with "I didn't catch that." I implemented a Dual-Signal Safety Net in turn_complete:
Signal 1 scans the user's transcript for intent keywords (recipe, pantry, cook, inventory) Signal 2 scans the AI's own output for betraying phrases like "let me check your pantry" when no tool was actually called
If either signal fires, the backend forces a _direct_delegate_fallback() — silently running the full orchestrator pipeline and pushing the result to the UI via WebSocket as a voice_results event. Users get their recipe regardless of ASR quality.
Asyncio deadlocks with the MCP stdio server. The MongoDB MCP server is a Node.js process communicating over stdio. Launching it inside a FastAPI async context inside Electron's subprocess tree created nested event loop conflicts. I had to carefully isolate MCP toolset initialization per-session and use asyncio.to_thread() for the CPU-bound PuLP solver to prevent event loop blocking during meal optimization.
Governance layer latency. Every agent write goes through policy evaluation before execution. Early versions had this as a sequential await chain: sanitize → evaluate → approve → execute. I collapsed the happy path (auto-approval below ₹250 and non-destructive) into a single asyncio.create_task() fire-and-forget, shaving ~400ms off the median receipt scan time.
Embeddings for Indian food vocabulary. text-embedding-004 doesn't natively know that "rajma" and "kidney beans" are the same thing, or that "methi" is "fenugreek leaves." I solved this by building a rich text blob per inventory item in build_inventory_document_text() that includes the category, sub-category, dietary flag, storage note, and manually curated use-case tags (protein grilling stir-fry curry, etc.) before embedding. This means "what do I have for a North Indian thali?" retrieves dal, paneer, and roti ingredients with high cosine scores even when those words don't appear in the query.
Accomplishments that we're proud of
voice agent where we can interact with it in live and it has access to every agent in the app so with that we can see what we are and the agent is speaking in the voice agent screen and if the text is too long example the items in inventory instead of saying 1 by 1 it shows in the the side ai workspace and also it can create recipe with asking kitchen chef agent
What we learned
ADK's transfer_to_agent is powerful but you have to be explicit. The orchestrator will hallucinate its own answers if the routing rules aren't tight. I ended up writing a strict keyword blocklist: the orchestrator is explicitly prohibited from answering inventory, recipe, or finance questions itself. Linear programming for meal planning is underrated. Most "AI meal planners" are just prompting an LLM. A proper PuLP solver gives mathematically guaranteed macro targets, handles impossible constraints gracefully with relaxed fallbacks, and runs in ~60ms.
MCP is the right abstraction for multi-agent tool sharing. Having all five agents share one read-only MCP toolset and a set of governed write wrappers meant I never had to duplicate MongoDB access logic across agents.
Voice UI requires fundamentally different output design. You can't read a 12-step recipe aloud. I learned to build a post-processing step that always extracts visual-renderable structured blocks from voice responses and dispatches them to the UI side panel, letting the voice be a summary while the screen shows the detail.
Log in or sign up for Devpost to join the conversation.