DietTrace

Inspiration

Logging your food is one of the first things people try when they want to eat healthier — and one of the first things they quit. The old way turns you into a database clerk: weigh every ingredient, search a difficult to use food tracking app, and pick the "right" entry from a dozen near-duplicates. The newer "just describe your meal to an AI" apps kill that friction, but they trade it for two worse problems: they hallucinate the numbers (there's no ground truth, so you never know when they're wrong), and they're static (they never learn how you actually eat). But why can't we have both accuracy and convenience?

What it does

DietTrace is an AI nutrition agent, not a chatbot. You type a meal in plain English — "two eggs, half an avocado, slice of toast", and the agent plans and acts across a deliberate tool pipeline: parse → search → estimate → math → log. The crucial rule is that the model never invents a number it can look up: nutrition comes from a deterministic lookup against USDA FoodData Central data, keyed by nutrient code, so the macros are exact and reproducible.

The one thing the agent can't look up is how much of each food you actually ate, so unless you say the amount, it has to estimate the portion. That portion guess is the biggest source of error in any tracker, and it's exactly what DietTrace learns to improve. Over time it adapts to your logging and portion style, while a deterministic gate guarantees it never regresses below a known baseline.

How I built it

• Agent runtime: Google's Agent Development Kit (ADK) with Gemini 3 on Vertex AI. Gemini handles parsing and orchestration; the lookups and math are deterministic code.
• Ground truth: USDA FoodData Central (Foundation + SR Legacy + a branded subset), CC0 public domain.
• Observability & evals: Arize Phoenix is the project's spine. Every run emits OpenInference/OTel spans; eval datasets and experiments live in Phoenix; the supervisor reads and writes them over the Phoenix MCP server.
• The self-supervision loop (the differentiator): DietTrace runs two agents. The food logging agent does the logging. The supervisor agent watches every meal, banks your corrections, and turns your confirmed meals into ground-truth dataset points (written to Phoenix via MCP). Once there's enough signal, it proposes a change to the food-logging agent, re-evaluates the new version against two datasets, your own meals and a fixed USDA base set, and a deterministic gate makes a simple go/no-go: it ships the change only if your meals improved and the USDA floor held. The agent decides what to try; real numbers on a held-out set decide whether it's good enough. It never grades itself.
• Surface: FastAPI backend + a Next.js / shadcn web UI, deployed on Google Cloud Run with Firestore for persistence. The right rail of the app is a live feed of the supervisor's work with feedback saved, dataset points added, re-tunes evaluated, so the whole loop is visible, not hidden.

Challenges I ran into

• Trustworthy numbers from a fuzzy input. The breakthrough was refusing to let the LLM produce nutrient values at all, making search a deterministic, reproducible USDA lookup and restricting Gemini to parsing and orchestration.
• Food matching is deceptively hard. "Chicken breast" should resolve to a plain cooked cut, not a deli roll; "half an avocado" needs an edible serving, not a pit-in weight. Tuning the ranking over the USDA set took real iteration.
• Portion estimation is the real issue. Once nutrition lookup is exact, almost all remaining error is the portion guess, which is precisely what the Phoenix eval loop measures and what personalization improves.
• Letting the agent adapt without letting it cheat. A naive "accept any feedback" agent games its own grader. The fix is a generator/verifier split, enforced by Arize: the agent proposes, but a deterministic gate decides, scoring every candidate as a Phoenix experiment on a held-out set it never trained on, plus the USDA floor.

What I learned

How to build an agent whose quality is measured, not asserted and using Arize Phoenix to turn "it feels accurate" into before/after experiments on named ground truth. I also learned how much leverage comes from the search/calculation split, how an MCP server can be the medium between two agents (one proposing changes, one verifying them), and that portion estimation, not nutrition lookup, is where accuracy tends to fall through.

What's next

• Photo logging and restaurant / mixed-dish estimation.
• A micronutrient-forward UI surfacing the full panel the database already carries.
• Apple Health sync and a React Native client.
• Letting the supervisor read recent traces over MCP to diagnose why a case regressed, not just that it did.

Built With

• arize-phoenix
• fastapi
• firestore
• gcp
• gemini-3
• google-adk
• mcp
• next.js
• openinference
• python