CartPilot

Say what you want to cook. CartPilot skips what's in your fridge, compares stores with live browser agents, and turns your leftovers into your next recipe. You choose what to buy; it never carts.

Comment

Inspiration

We've all been there: halfway through baking a cake, you realise you're out of eggs. Or you buy a 1kg bag of cocoa powder when the recipe only needed 40g, and the rest sits in the cupboard for a year. Comparing groceries is surprisingly hard too. Stores sell the same item in different pack sizes ("500 g", "6 x 50ml", "12's"), so the sticker price is misleading, and look-alike products like "cocoa drink mix" vs "cocoa powder" are easy to mix up.

I wanted an assistant that does the tedious part (working out what you need, checking what you already have, and comparing stores) while leaving every decision to you.

What it does

CartPilot turns a cooking goal into a price-compared shopping list and a step-by-step recipe:

  1. Tell it what you want to make, e.g. "bake a chocolate cake for 6". The AI suggests 2–3 recipes and writes the ingredient list, scaled to your servings.
  2. Snap a photo of your fridge. The AI checks which recipe ingredients you already have, with a confidence level for each. Unsure items are marked Please check instead of being assumed.
  3. Approve the list. Nothing automated happens before this.
  4. Watch AI agents search multiple stores in parallel, live, in a grid of browser screens with an activity log and running totals.
  5. Get the best product for each ingredient, chosen by price per unit and smart pack size, with alternatives you can swap in.
  6. See a Leftovers panel with recipe ideas that use up what you'll have extra of.
  7. Follow the full recipe. CartPilot gives beginner-friendly, numbered cooking steps with prep and cook times, using the quantities scaled to your servings.

CartPilot never adds to cart, logs in, or buys anything. The final output is a list of product pages, and you decide what to buy.

How I built it

  • Backend: Python + FastAPI, with a WebSocket streaming live browser screenshots and progress to the UI.
  • AI: Google Gemini (free tier) for recipe suggestions, ingredient lists and cooking steps, fridge photo recognition, product match review and leftover ideas. Every AI response uses JSON mode and is validated with Pydantic. The ingredients and steps come from a single AI call, so they always match and stay within the free tier.
  • Browser agents: Playwright runs a pool of parallel browser pages that search each store.
  • Mock grocery stores: I built 3 fake grocery sites with different layouts, brands, prices and messy pack-size formats, so the demo is reliable and doesn't depend on scraping real stores.
  • Frontend: Plain HTML, CSS and JavaScript, no build step.

To pick the best product, I convert every pack size into a standard unit and compare cost per 100g/100ml/piece:

$$\text{price per unit} = \frac{\text{price}}{\text{pack size in standard units}}$$

Then CartPilot chooses the cheapest combination of packs that covers what the recipe needs, not just the cheapest pack, so you don't overbuy.

Responsible AI

  • Human in control: automation only starts after the user approves, and a Stop button halts every agent instantly.
  • Safety guard in code: the agent physically cannot click anything like "add to cart", "checkout", "pay" or "login", enforced by a guard with its own tests, not just a prompt.
  • Confidence flags: I didn't want to blindly trust the AI. Its confidence is combined with rule-based checks (e.g. product name doesn't match, size can't be read, different product form), and low-confidence items must be confirmed by the user.

Challenges I ran into

  • Parsing pack sizes was harder than expected: "6 x 50ml", "1.5L", "12s" and "Value Tin" all need different handling. I wrote extensive unit tests to catch bugs.
  • Look-alike products: one store's search ranked "Cocoa Powder Drink Mix" first for "cocoa powder". I added rule-based matching so the agent picks the real product.
  • Real store automation: real grocery sites use login walls and bot detection. Instead of trying to bypass them, I built mock stores behind a common adapter interface, so real stores can be plugged in properly later through official APIs.
  • Live view: streaming screenshots from several parallel browsers without lag took some tuning.

Accomplishments that I'm proud of

  • A complete end-to-end flow from "I want to bake a cake" to a price-compared product list and a recipe ready to follow.
  • Watching multiple stores being searched at once, live.
  • A safety design where the AI helps but can never spend your money.
  • A test suite of nearly 400 tests covering units, matching, safety and the agent.

What I learned

  • How to combine LLMs with plain rule-based code: the AI is great at understanding goals and photos, but simple maths and rules are more reliable for prices and units.
  • How to run and coordinate parallel browser automation with Playwright and asyncio.
  • That responsible AI is mostly about design choices: approval steps, confidence flags and hard limits in code.

What's next for CartPilot

  • Connecting to real grocery retailers through official APIs or partnerships.
  • Dietary filters (halal, vegetarian, eggless, nut-free).
  • Weekly meal planning that shares ingredients across recipes to cut waste further.
  • Expanding beyond cooking, like building a PC or planning a party.

Built With

Share this project:

Updates

Submission history