SmartCart AI - A Kitchen Operating System

AI-powered kitchen operating system that turns your pantry into a week of meals, a smart budget, and the cheapest store run nearby.

Comment

Inspiration

KitchenOS began life as SmartCart AI, originally built for the Google Cloud Rapid Agent Hackathon and later rebuilt for the Alibaba/Qwen Cloud Hackathon. Both versions solved the same core problem — grocery shopping shouldn't require a spreadsheet, three browser tabs, and a calculator — but both still leaned on the user to steer the conversation: type a request, get a response, ask a follow-up.

For Build Week, we asked a different question: what if the agent didn't need to be steered at all? What if, instead of a chat you have to drive, you get a true autonomous agent — one that looks at your pantry, decides what to cook, builds the list, checks the budget, and compares real nearby stores, all without requiring any user feedback mid-flow?

That shift — from a conversational assistant to a genuinely autonomous agent — is what KitchenOS is built around.

What it does

KitchenOS offers two distinct agent modes, depending on how much the user wants to specify up front:

Plan My Week is the fully autonomous path. No preferences required — the agent looks at what's in your pantry and what's realistic on a budget, and independently chains five tools into a single run to produce a complete plan:

  1. Get Pantry — checks what you already have on hand
  2. Generate Weekly Meals — proposes a set of meals built around your pantry and dietary preferences
  3. Build Shopping List — subtracts what you already own, so you never buy duplicates
  4. Weekly Budget Planner — optimizes the list against your stated budget and flags substitutions
  5. Recommend Best Store — compares your real shopping list against actual nearby grocery stores (using location + Places data) and returns a live, itemized price comparison so you can see exactly which store is cheapest and by how much

The result: a full week of meals, a combined shopping list, a nutrition report with AI-generated feedback, and a side-by-side store comparison — all before you've bought a single item. You review it, hit Approve & Save, and it's saved to your profile.

Optimize My Cart is the counterpart for when the user does have preferences to give. If you tell the agent your meal preferences, it generates Optimize My Cart instead — running the same pantry, budget, substitution, and store-comparison tools, but shaped around what you actually said you want to eat, rather than deciding independently.

The chat interface makes the agent's reasoning visible rather than hidden. When a user clicks "Plan meal from my pantry" in chat, they can see the exact agent tool being invoked in real time — via AgentTrace, a transparent, step-by-step trace of every tool the agent called and what it found. Nothing about the plan is a black box.

Beyond these two flows, KitchenOS also includes:

  • Receipt Upload — extracts real prices from an uploaded grocery receipt, which future price comparisons can then use instead of estimates
  • GPT-5.6 with automatic Gemini fallback — if the primary model is unavailable or fails, the app fails over to Gemini so the experience never breaks

How we built it

This is the third hackathon iteration of the same underlying codebase — Google Cloud Rapid Agent Hackathon, then Alibaba/Qwen Cloud Hackathon, now Build Week — and each rebuild pushed the architecture further rather than starting over. That iteration shows: this submission isn't a weekend prototype; it's a codebase that has been rebuilt against three different AI stacks (Gemini/Vertex AI, then Qwen Model Studio, now GPT-5.6 with Gemini fallback) and hardened each time.

The autonomous agent features built specifically for this Hackathon — Plan My Week, the rebuilt store-pricing pipeline, and the approve/save flow — were implemented using OpenAI Codex as the coding agent, with every change tested and verified end-to-end against the live deployed site before merging. GPT-5.6 is the model that powers the app's own AI reasoning at runtime (meal generation, budget optimization, nutrition feedback), with automatic Gemini fallback.

  • Frontend: React, deployed to Firebase Hosting, with a multi-target setup so this Build Week submission, an earlier Qwen-based submission, and an XPRIZE submission can all ship independently from the same codebase
  • Backend: FastAPI on Google Cloud Run, containerized and deployed via Cloud Build / Artifact Registry
  • AI layer: GPT-5.6 as the primary model with a Gemini fallback path (generate_primary_or_fallback), so a single provider outage doesn't take down the app
  • Data: Firestore for user-scoped pantry, meal plans, and receipts; MongoDB-backed caching for fast repeat lookups
  • Auth: Firebase Auth with Google Sign-In
  • Store data: a real store-finder pipeline that filters nearby results down to actual grocery stores (excluding restaurants, gas stations, etc.) and prices the generated shopping list against each one to produce a genuine per-store basket comparison

Challenges we ran into

The hardest bug wasn't a crash — it was a silent data mismatch. Our /plan-my-week/approve endpoint kept returning [object Object],[object Object] in the UI, which turned out to be a FastAPI validation error array being rendered raw instead of parsed. Fixing the display was easy; finding the real cause took longer: the Pydantic model expected a flat {name: description} dict, but the actual meal-suggestion data was a list of {name, reason} objects. We had to trace the true shape all the way from the AI's generation step through to the save layer before the fix could be correct instead of just quiet.

A second, subtler bug: our first version of the store-recommendation tool found nearby places by location only — which meant a demo run once recommended a gas station and a restaurant as grocery options, both showing $0.00 because nothing had actually been priced. We rebuilt that pipeline to filter to real grocery stores and price the actual shopping list against each one, so "cheapest store" is now a real, verifiable number instead of a placeholder.

We also hit a string of deployment environment issues in the final hours — mismatched GCP/Firebase project IDs, a legacy gcr.io permission gap that Artifact Registry cleanly solved, and a stale Cloud Run deploy that briefly had production running yesterday's backend against today's frontend. Each one taught us to verify deploys by hitting the live URL directly, not by trusting that "no errors" meant "actually updated."

Accomplishments that we're proud of

  • A genuinely multi-agent pipeline where each tool call is visible and inspectable via AgentTrace, not hidden behind a single opaque prompt
  • Real store price comparison — not simulated, not hardcoded — built from actual nearby grocery stores and an actual generated shopping list
  • A resilient AI layer that gracefully fails over between GPT-5.6 and Gemini
  • Catching and fixing a data-shape bug that could easily have shipped silently broken

What we learned

That the gap between "the API returned 200" and "the feature actually works" can be enormous — and that the only real proof is testing the literal end-user flow, end to end, on the live deployed site. We also learned to distrust our own assumptions about data shape; several of tonight's hardest bugs came from guessing a field's structure instead of printing it and looking.

What's next for KitchenOS

  • Wire receipt-uploaded prices more deeply into the store comparison, so verified prices increasingly replace estimates over time
  • Expand Plan My Week with dietary constraint presets and multi-week planning
  • Bring the Compare Nearby Stores view into the shared Store Prices tab for full item-by-item breakdowns
  • Add route optimization so approved plans can suggest the most efficient multi-stop shopping trip

Built With

Share this project:

Updates