Bakery Partner Co-Pilot

Inspiration

The idea came from watching my friends who run solo bakeries. I kept hearing the same thing from them: keeping track of inventory and revenue was genuinely complicated, especially for the ones who didn't come from a business background. It wasn't that they didn't care about the numbers, it was that the tools for tracking them assumed a kind of spreadsheet fluency that wasn't the point of why they started baking in the first place.

I started by talking through the problem with Gemini, using it to brainstorm and pressure-test what a "collaborative partner" for a solo baker should actually look like, before I ever wrote a line of code. That early conversation is where the framing solidified: this shouldn't be a chatbot that answers questions about baking. It should be a partner that takes over the operational mental load, the way a good assistant manager would, without asking the baker to think in numbers or systems they never signed up for.

What it does

Bakery Partner Co-Pilot is a conversational operations assistant for a solo baker. You talk to it, by typing, speaking, or dropping in a photo or PDF of an order note or invoice, and it handles the operational side of running the business:

• Inventory: tracks ingredients, auto-deducts stock when something is baked, flags low-stock and expiring items, and can add new ingredients or recipes on the fly.
• Orders: manages a full order lifecycle (pending to fulfilled to paid or cancelled), tracks who owes what, and asks for clarification when a request is ambiguous, like when two open orders match the same customer name.
• Financials: generates weekly and monthly revenue and cost summaries, suggests pricing from ingredient cost plus target margin, and tracks non-ingredient expenses.
• Planning: builds a shopping list from planned bakes, and a production plan straight from open orders.
• Memory: remembers standing preferences ("don't bake tarts on rainy days") and per-customer notes (allergies, habits) that shape its behavior in every future session, not just within one chat.

A live dashboard shows open orders, this week's money snapshot, and inventory alerts, pulled directly from Firestore with no LLM call required.

How we built it

This project was really an exercise in HCI and UX thinking as much as it was an engineering one. I built it solo. The core design question wasn't "what can the model do," it was "what should the baker never have to think about, and where does the agent need to stop and ask instead of guess." Most of my time went into shaping that interaction model: when the agent should confirm before acting, how it resolves ambiguity through conversation instead of demanding technical detail, and how memory should carry across sessions so the baker never has to repeat themselves.

I used Gemini early on to brainstorm and define the problem space and the shape of a "collaborative partner" agent. Once the design was clear, I used Claude Code to build it out: a persistent Gemini 3.5 Flash chat session (via Vertex AI) with 21 registered tool functions that read and write to Cloud Firestore, with Streamlit as the frontend for the chat panel and live dashboard.

Architecturally, Gemini and Firestore never talk to each other directly. A single backend module owns the chat session and mediates every tool call, so all state changes go through one auditable path. Voice input is transcribed via a separate one-shot Gemini call before rejoining the normal tool-calling flow, and the dashboard reads Firestore directly for instant refreshes without waiting on the model.

It's deployed on Cloud Run with a dedicated, least-privilege service account (Firestore and Vertex AI access only, not the broader credentials used locally), and it scales to zero when idle. A 50-test pytest suite runs against an in-memory Firestore fake, so the whole suite runs offline with no GCP credentials needed.

Challenges we ran into

• Model upgrades aren't just a name swap. Moving from gemini-2.5-flash to gemini-3.5-flash silently broke several tools. The newer model validates function-calling schemas more strictly and rejected bare list type hints that 2.5 had tolerated without complaint. I also learned gemini-3.5-flash is currently only served on Vertex AI's global endpoint, not regional ones.
• Revenue double-counting was a real risk once the agent could both close an order and log a sale independently. Rather than trying to hard-block every path, which risked false positives, I split the responsibilities clearly in the system instructions and made the order-paid transition idempotent, so even a confused model can't double-charge the same order.
• Ambiguous instructions caused confidently wrong behavior, not just missing ones. Requiring an expiration date before restocking an existing ingredient was correct, but the model generalized that rule too broadly and started demanding a placeholder expiration date when adding a brand-new ingredient, where no such requirement exists. The fix was scoping precisely which tool a rule applied to, not writing the rule more carefully.
• Institutional Google Cloud orgs can block public Cloud Run access by policy. My project sits under a Cornell-managed GCP organization with a domain-restricted-sharing constraint, which silently blocks granting allUsers access, independent of anything in the deploy configuration. I verified the deployment is genuinely live via an authenticated request instead, and documented the workaround for anyone deploying under a similar institutional org.

Accomplishments that we're proud of

• Getting the agent to genuinely ask instead of guess. It never invents an expiration date, never fabricates revenue from a list price, and won't act on something read from a blurry photo without confirming first. That behavior held up consistently across testing, not just in the happy path.
• Building a real memory layer, not just a longer context window. Standing preferences and per-customer notes persist across sessions and actually change how the agent behaves later, which is a meaningfully different bar than most chatbot demos clear.
• Shipping a genuinely multimodal input path (text, voice, photo, PDF) that all funnels into the same tool-calling flow, so the baker never has to think about which "mode" they're supposed to use.
• Keeping the whole thing testable and safe by design: 50 tests running fully offline against an in-memory Firestore fake, and a least-privilege service account for the live deployment instead of reusing broader local credentials.
• Getting a real, working Cloud Run deployment end to end, including diagnosing and documenting an institutional org policy blocker that had nothing to do with my own configuration.

What we learned

Building a "collaborative partner" agent is less about the model and more about designing for ambiguity: where the agent should ask instead of assume, how narrowly a system instruction needs to be scoped so it doesn't leak into contexts it wasn't meant for, and how to make state transitions safe even when the model occasionally gets confused. Using two different AI tools for two different phases, Gemini for problem definition and Claude Code for implementation, also clarified for me how much of good agent design happens before any code gets written: in the UX decisions about what the agent should track, ask, and remember on its own.

What's next for Bakery Partner Co-Pilot

• Get it in front of the friends who inspired it and iterate on real usage, not just my own testing.
• Support multiple bakers/tenants instead of a single-baker Firestore instance, so this could actually be offered as a shared tool.
• Add lightweight forecasting: using order history to suggest what to bake ahead of a busy weekend, rather than only reacting to open orders.
• Explore a public-facing deployment path that doesn't depend on an institution's Google Workspace org, so a real baker could try it without needing a GCP account at all.
• Tighten the voice input experience further, ideally handling multi-turn voice conversations, not just single transcribed utterances.

Built With

• firestore
• gemini
• python
• sdk