Hold the Order

Hold the Order is the autonomous dinner agent that turns food you already bought into one meal your family will actually make, saving time, money, and groceries before takeout wins. has context menu

Comment

Inspiration

Hold the Order began with a contradiction in our own home: we would buy groceries, fill the refrigerator and freezer, and still order takeout because deciding what to make felt harder than ordering food. The problem was not a lack of recipes. It was the moment when everyone was tired, schedules had changed, ingredients were aging, nobody wanted extra dishes, and twenty recipe suggestions felt like twenty more decisions. We wanted an agent that could step into that exact moment, understand both the available food and the people eating it, and carry dinner forward. That led to one defining principle: one realistic answer is more useful than twenty recipes.

What it does

Hold the Order is an autonomous household dinner agent that turns food a family already bought into one meal they will actually make. A household begins with a guided refrigerator or freezer scan. Gemini 3.5 Flash identifies visible food, organizes it by storage area, and marks uncertain items for review. The system never claims it can see through an opaque container or determine food safety from a photograph. Instead, it offers likely guesses, recent household leftovers, voice entry, and an open-the-lid rescan. The agent combines verified food evidence with explicit household information such as:

  • Allergies and dietary restrictions
  • Food preferences
  • Arrival times
  • Energy level
  • Cleanup tolerance
  • Available cooking time
  • Ingredients that should be used soon It then recommends one practical dinner instead of presenting a feed of recipes. The Tonight experience immediately explains:
  • What to make
  • How long it should take
  • How that compares with waiting for delivery
  • Which refrigerator or freezer items it uses
  • How much takeout spending could be avoided
  • Why the meal fits the household
  • What to do next Hold the Order continues working after making the recommendation. It can create a shared cooking checklist, assign tasks to household members, schedule timers, guide a novice cook step by step, recover when a task is declined or an ingredient is missing, and leave an action in the Agent Inbox when a notification cannot be delivered. After dinner, the household records one simple outcome:
  • Made it
  • Adapted it
  • Ordered out
  • Leftovers tomorrow That outcome closes the workflow, reconciles confirmed ingredients, updates estimated savings and rescued-food impact, and creates a visible, correctable memory for future dinner decisions. Additional experiences include refrigerator and freezer history, a seven-night Dinner Runway, Leftover Passports for covered containers, family profiles, estimated weekly and monthly savings, takeout comparisons, guided cooking, cross-device household synchronization, and Judge Mode for inspecting the complete autonomous workflow. ## How we built it Hold the Order is a responsive React 19 progressive web application built with vinext and hosted on Google Cloud Run. The public web service supports mobile and desktop experiences while providing a same-origin server proxy so private credentials never enter the browser bundle. The autonomous agent runs as a separate Node.js service on Cloud Run. The Google Cloud architecture includes:
  • Gemini 3.5 Flash on Vertex AI for multimodal refrigerator and freezer analysis, structured inventory generation, dinner planning, and selective plan repair
  • Gemma 4 on Vertex AI for protected, open-ended Countertop Mode cooking questions
  • Veo 3.1 Fast for optional Food Future and Dinner Trailer videos requested by the user
  • Cloud Run for the responsive web application and autonomous agent runtime
  • Cloud Scheduler for recurring household policy evaluations
  • Pub/Sub for durable background dinner events, retry delivery, and dead-letter handling
  • Cloud Tasks for cooking timers, reminders, and asynchronous media jobs
  • Firestore Native for household profiles, scan history, workflows, outcomes, tasks, devices, and correctable memory
  • Cloud Storage for private generated media that expires automatically
  • Secret Manager and IAM for scoped credentials and service access
  • Cloud Logging for operational telemetry and workflow evidence
  • Web Push for timely dinner interventions, with an Agent Inbox fallback Every workflow receives a durable ID connecting the original event, model execution, meal plan, household tasks, notification receipt, response, outcome, recovery workflow, and memory used during the next run. Pub/Sub deliveries use a create-only Firestore event record as an idempotency lock. If the same event is delivered twice, the second delivery is acknowledged without producing another meal plan, notification, or task list. We separated the product into two layers:
  • Everyday Mode focuses on Tonight, Fridge + Freezer, Family, and History.
  • Judge Mode exposes model mode, decision thresholds, workflow classifications, execution stages, delivery status, retries, recovery lineage, outcomes, and memory provenance. This keeps the family experience calm while making the technical architecture inspectable. ## Challenges we ran into ### Making computer vision useful without overstating it Refrigerators are difficult visual environments. Items overlap, labels face the wrong direction, food sits behind cloudy plastic, and leftovers may be completely hidden inside containers. Early results were technically cautious but not useful enough because they described containers instead of helping the household identify the food. We redesigned the experience to separate visual observation, model inference, and human confirmation. Gemini can offer bounded food guesses, but the interface preserves whether an answer came from vision or from a household member. ### Keeping inventory trustworthy over time A refrigerator scan begins aging as soon as someone eats, moves, discards, or purchases something. Requiring a complete scan every day created too much friction. We added multi-day snapshots, changed-shelf scans, receipt additions, use and discard actions, automatic evidence decay, and targeted refresh requests. The agent asks for a new view only when the existing evidence is no longer strong enough to support an action. ### Proving autonomy instead of simulating it A polished button connected to a model is not an autonomous agent. We built scheduled evaluations, durable events, idempotency protection, retries, push delivery, recovery workflows, outcome capture, and memory lineage. The application distinguishes between a user-requested plan, a judge replay, an autonomous recovery, and a threshold-qualified autonomous intervention. ### Synchronizing private household state across devices Phone-to-desktop continuity could not depend on browser local storage. Google sign-in restores a private household identity, each connected device receives a separate revocable credential, and every Firestore-backed route checks household ownership before reading or changing data. ### Balancing model quality, latency, and cost Detailed multi-view vision can become expensive if every shelf is repeatedly analyzed. We created a staged vision pipeline that handles obvious regions first, escalates ambiguous areas, deduplicates repeated items across views, avoids rescanning unchanged evidence, reserves household quota before paid calls, and records model calls, duration, tokens, and estimated cost. ### Making a feature-rich product feel calm The system includes scans, family constraints, savings, tasks, timers, notifications, cooking guidance, memory, recovery, and technical proof. Showing everything simultaneously made the product feel like a case study instead of a nightly utility. Multiple design passes helped us establish a simpler hierarchy: one dinner decision, a few supporting reasons, and optional evidence behind progressive disclosure. ## Accomplishments that we're proud of We are proud that Hold the Order completes an end-to-end agent workflow instead of stopping at generated text:
  • It observes household-approved refrigerator and freezer evidence.
  • It requests clarification when evidence is uncertain.
  • It combines food with real household constraints.
  • It chooses one achievable dinner.
  • It dispatches the workflow asynchronously.
  • It coordinates people, tasks, timers, and delivery.
  • It records what actually happened.
  • It applies a correctable memory during the next run. We are also proud of the product's honesty. Covered food remains uncertain until a person confirms it. Visual freshness is never presented as a food-safety measurement. Savings remain estimated until an outcome is recorded. A failed notification remains failed and falls back to the Agent Inbox. A duplicate event is rejected. A manual plan is never relabeled as an autonomous intervention. The same workflow ID connects the customer experience with Google Cloud evidence, including Pub/Sub delivery, Cloud Run execution, Firestore state, household actions, outcomes, and memory. Finally, we built one shared household product across mobile and desktop. A refrigerator scan, family update, task response, cooking session, or recorded outcome can continue on another signed-in device. ## What we learned The biggest lesson was that useful agents are measured by completed decisions, not by the amount of text they generate. The most valuable moment in Hold the Order is not when Gemini describes a refrigerator. It is when the family receives one credible plan and cooking becomes easier than opening a delivery app. We also learned that uncertainty can become a product feature when it is handled honestly. Asking one precise question can create more trust than producing a confident but unsupported answer. Human corrections are not failures of the agent. They are evidence that makes the next action safer and more personal. Persistent memory only matters when users can understand and correct it. Hold the Order ties memory rules to reported outcomes and source workflow IDs instead of inferring preferences from private conversations or presenting invented confidence scores. Autonomy also requires graceful failure. Idempotency, retries, inbox fallback, deterministic model fallback, evidence decay, and linked recovery workflows are less visually dramatic than a generated meal, but they are what make the system dependable. Finally, we learned that technical sophistication should be inspectable without becoming the everyday interface. Families need a calm answer. Judges and developers need a flight recorder. Both can exist in the same product when they are intentionally separated. ## What's next for Hold the Order The next major milestone is a larger longitudinal household pilot. We want to measure meal acceptance, scan corrections, completed outcomes, food rescued, takeout avoided, and recommendation quality over multiple weeks without presenting invented statistical confidence. We also plan to:
  • Complete and film the full physical-iPhone workflow from scheduled evaluation to notification, recorded outcome, and memory-adjusted next run
  • Add optional Google Calendar arrival windows
  • Add grocery receipt parsing
  • Add Siri Shortcuts and Home Assistant signals with explicit permission
  • Make changed-shelf detection and inventory reconciliation faster
  • Improve container identification through household leftover history and reusable QR passports
  • Explore a native iOS application if PWA notification limitations materially affect adoption
  • Add retailer and appliance integrations without giving the agent authority to purchase food
  • Expand freezer-aware weekly planning while remaining honest about evidence age and future availability
  • Continue improving cost, latency, accessibility, security, and household privacy controls Our goal is not to build another recipe application. It is to create a trusted household decision agent that notices what a family already has, understands what tonight feels like, and quietly carries dinner across the finish line before takeout wins.

Built With

  • cloudrun
  • cloudscheduler
  • cloudstorage
  • cloudtasks
  • firestore
  • gemini3.5flash
  • gemma4
  • google-cloud
  • googlesdk
  • veo3.1fast
  • vertex
Share this project:

Updates