Gemini FoodLog

Problem

Food and symptom diaries can help people investigate possible triggers, but manual logging is burdensome and incomplete. Existing approaches still ask the person to photograph a plate, scan packaging, weigh ingredients, search for a dish, or type a description. FoodLog starts with a stricter requirement: the household should not have to change how it cooks.

Real kitchen evidence is messy. A hand obscures an ingredient, a label faces away, several low-quality frames belong to one event, leftovers resemble a new meal, and a pet on the counter is activity but not cooking. The useful unit of work is therefore not one image classification. It is a background workflow that gathers evidence, maintains a revisable hypothesis, retrieves only relevant context, decides when uncertainty deserves a question, and persists the result.

Solution

FoodLog turns passive, ordinary kitchen observations into a revisable food journal. It groups images over time, retrieves only relevant account context, uses an agent to form an evidence-linked hypothesis, asks a focused question only when it would materially help, and preserves corrections as future household knowledge.

Key Features

A signed-in phone camera, the Python capture client, or a revocable physical camera credential sends ordinary JPEG or PNG observations through one API.
The API enforces the account's image entitlement, stores the immutable image privately, persists metadata, and publishes a durable event.
Independent Cloud Run workers group captures into revisioned kitchen events and run a Google ADK agent with Gemini 3.6 Flash on Vertex AI.
The agent receives ordered private images plus bounded, account-scoped tools for relevant purchases, temporary context notes, household knowledge, and prior corrections. It emits an evidence-linked structured hypothesis rather than an unsupported label.
The web application shows the real image sequence, tentative or confirmed meal, uncertainty, observations, alternatives, context used, and complete revision history. The user can confirm it, correct a meal or component, or discard an event as not cooking.
Corrections remain in immutable history and can create versioned household knowledge. Separate longitudinal questions let the agent ask whether a repeated pattern is real without turning unidentified-meal questions into a generic inbox.
Authenticated forwarded supermarket order and invoice emails can become provenance-preserving purchase evidence; the current prototype is validated against one retailer format. A person can also add time-bounded context such as an unusual ingredient expected tomorrow.
The account-data page exposes the records FoodLog holds and supports a private, temporary export. Stored evidence is tenant-scoped; browser clients cannot read Firestore or Cloud Storage directly.

The food timeline is intended to support later, cautious exploration of possible food and symptom associations. FoodLog does not diagnose conditions or claim causality. Calories are secondary to building a low-effort longitudinal record.

Why This Matters

People investigating food-related symptoms need longitudinal evidence, but the burden of manually maintaining a diary makes that evidence sparse precisely when it matters. FoodLog aims to make the journal emerge from ordinary life without asking the household to stage ingredients, scan products, or remember to log every meal.

FoodLog completes a messy personal workflow instead of returning a chat answer. It runs asynchronously, transforms unstructured multimodal observations into durable state, calls bounded retrieval tools, handles retries and partial failures, chooses whether human input is worth requesting, and updates a real product record. Its core action is maintaining the journal over time.

Architecture

Agent and model: Google ADK, Google Gen AI SDK, Gemini 3.6 Flash, and the Vertex AI eu multi-region.
Google Cloud: six scale-to-zero Cloud Run services, Pub/Sub with isolated subscriptions and dead-letter paths, Firestore, private Cloud Storage, Artifact Registry, Secret Manager, Cloud Logging, Cloud Monitoring, and an App Engine inbound-mail gateway.
User surface: React, TypeScript, Vite, Firebase Hosting, and Firebase Authentication.
Capture clients: protected browser manual/motion capture, a locked Python client for fixtures or webcams, and deployed firmware 0.2.0 for the Freenove FNK0085 ESP32-S3 camera board.
Delivery and operations: Terraform, keyless GitHub Workload Identity Federation, immutable container digests, protected deployment approval, bounded model-spend reservation, and source-controlled operational runbooks.

The production architecture diagram shows the public, asynchronous, agentic, and tenant-data boundaries. The deployment guide contains the clean-checkout and cloud release procedure.

How We Used AI

Gemini 3.6 Flash receives an ordered, bounded set of private kitchen images and works through a Google ADK tool loop. The account-scoped tools retrieve only the purchase evidence, temporary context, household knowledge, and prior corrections relevant to the event. The model returns a validated structured hypothesis with literal observations, alternatives, uncertainty, evidence references, and at most one focused question. Deterministic application guards reject unsupported certainty and preserve the original result plus every correction as immutable history.

Gemini 3.1 Flash TTS Preview produced the modular English narration for the submission video. Two bounded Veo 3.1 Lite generations provide the opening visuals, and Lyria 3 Pro Preview provides its instrumental score. These models are integrated into the reproducible demonstration-media pipeline rather than the production food-inference path; their prompts, exact costs, hashes, and verification evidence are source-controlled.

Authorized data sources

kitchen images deliberately uploaded by the account's browser or camera;
user corrections, pattern answers, temporary context notes, and explicitly taught household knowledge;
authenticated online-supermarket order and invoice evidence forwarded to an opaque per-account address;
application-generated capture, grouping, inference, purchase, feedback, accounting, and revision metadata.

All evidence is tenant-scoped. Gemini receives only the bounded evidence selected for one account and one event, and operational traces are redacted.

Findings and learnings

Time changes the problem. A blurry ingredient can be ambiguous in one frame but useful when combined with what appeared immediately before and after it. Durable event grouping must precede meal reasoning.
Uncertainty needs product semantics. A weak best guess, a genuinely unknown event, and non-cooking activity require different actions. A blanket “Looks right” button teaches the wrong thing.
Questions belong with their evidence. Event-specific ambiguity stays on the matching timeline card. A separate agent-observation surface is useful for longitudinal hypotheses such as a recurring weekday breakfast pattern.
Learning needs provenance and scope. “The next white meat may be duck” is temporary context; “this household often cooks steak in this basket” is a revisable household belief. Confirmation must not silently broaden either.
Retries are normal, not exceptional. Images, Pub/Sub messages, model calls, feedback, and email all need stable identities, immutable revisions, bounded retry behavior, and inspectable failure states.
Cost controls must exist before autonomy. Public accounts are capped at 25, public trials at 200 accepted images, Cloud Run at one instance per service, and every Gemini workflow reserves against a DKK 400 application ceiling before calling the provider.
Privacy changes the architecture. Account identity is derived from verified authentication or server-owned events, object keys are server-generated, direct browser database access is denied, and cross-tenant negative tests are part of the release evidence.

Known Limitations

Manual phone capture and the physical ESP32-S3 camera work end to end, including authenticated motion capture and encrypted offline recovery. Multi-hour phone and multi-day ordinary-kitchen endurance testing remain outstanding.
The live Gemini path has passed real-image and bounded synthetic production tests. Prompt quality and question frequency still need evaluation against a longer ordinary-kitchen dataset.
Purchase-email ingestion is currently validated with one online supermarket; broader retailer coverage remains to be tested.
The MVP records food, not symptoms. It can support later comparison with symptom information kept elsewhere, but it is not a medical device and does not make diagnostic or causal claims.