Sous: The NutriAgent

Sous is a multimodal agent powered by Gemini 3. It connects visual context with deep reasoning to solve decision paralysis. It observes, thinks, and acts to help the user live a healthier life.

Comment

NutriAgent: The Cognitive Sous-Chef

Stop logging. Start eating. NutriAgent is a web-based health agent that turns goals into an executable daily loop—Recipe → Fridge → Shopping → Meal Log → Next Recommendation—and extends it with Workout Logs so nutrition and training decisions reinforce each other.

Inspiration

Healthy living usually doesn’t fail because people lack motivation. It fails because the decision workload is constant:

  • “What should I eat that fits my plan?”
  • “Do I have the ingredients?”
  • “If not, what should I buy?”
  • “How does this interact with my workout today?”
  • “And do I really need to manually log everything?”

Most apps are passive trackers: they demand perfect manual input and return generic charts. We wanted a Sous-Chef: an agent that can reason over context, maintain state (what you have, what you ate, what you trained), and push you forward with the next best action.

What we built

NutriAgent (codename Sous) is centered around a plan-driven loop with five practical capabilities:

1) Plan → Recipe (Goal-aligned reasoning)

Sous starts from the user’s health plan and constraints (e.g., high protein, weight loss, low sodium, time limits) and proposes recipes that are feasible and goal-consistent—not just “healthy-sounding.”

2) Menu → Best Choice (Context-Aware Analysis)

At a restaurant, Sous can analyze menus and recommend the single best dish based on your profile (e.g., high protein, low sodium, weight loss). It also flags misleading options—“healthy-looking traps” like salads that are calorie-dense due to dressings, toppings, or sauces—so users don’t accidentally sabotage their plan while trying to be “good.”

3) Fridge ↔ Recipe Execution Loop (Inventory → Shopping → Meal Log)

Instead of relying on fridge photo recognition (not fully tested yet), NutriAgent focuses on a stateful closed loop that converts recommendations into real execution:

  • Recipe / nutrition facts → Fridge: When the agent proposes a recipe, it logs structured ingredients and nutrition facts directly into the fridge inventory model.
  • Fridge → Recipe: Based on the current fridge state, Sous generates feasible recipes aligned with the health plan.
  • Missing items → Shopping list: If a recipe requires ingredients not present in inventory, Sous automatically adds them to a shopping list—turning “planning” into a concrete next step.
  • Meal log → Plan feedback: Once the user eats, the meal is logged, macros are updated, and the agent uses this updated state to improve the next recommendation.

This creates a practical cycle: Plan → Recipe → Inventory → Shopping → Meal Log → Better Next Recommendation …with minimal manual input and a persistent state that keeps the agent grounded.

4) Natural-language Logging → Structured Actions (Agentic automation)

Users don’t need to fill forms. They can say things like: “I ate the salmon bowl.” Sous converts intent into structured JSON actions to update the database (meal log, inventory adjustments, macro tracking), keeping the experience fast and human-friendly.

5) Nutrition + Workout Guidance (Joint recommendations)

NutriAgent isn’t only about food. It also incorporates workout logs to generate recommendations on both sides:

  • nutrition suggestions that match training intensity (e.g., recovery-focused meals after a hard session)
  • workout suggestions that fit nutrition status and goals (e.g., adjust volume if intake is low, prioritize consistency when user is fatigued)

The goal is to treat health as one system: training and eating should co-plan, not live in separate apps.

How we built it

We designed NutriAgent as a Cognitive Interface over a headless, tool-driven backend.

Frontend: Flutter Web (The control panel)

  • Web UI with flows for recipe proposals, menu analysis, confirmation, and logging
  • A human-in-the-loop proposal card pattern: the agent proposes structured actions → the UI renders them as editable cards → the user confirms → then it commits
  • UX designed to make the agent feel collaborative, not opaque

Backend: FastAPI + Docker (Memory and state machine)

  • Persistent state: user profile, health plan, fridge inventory, shopping list, meal logs, workout logs
  • Deterministic endpoints the agent calls as tools (log meal, update inventory, add shopping items, log workout, recalc macros)
  • Clean separation: the agent reasons, but the backend is the source of truth

Agent layer: Gemini 3 (Reasoning engine)

  • Multimodal + low-latency reasoning
  • Structured output (JSON) for tool execution
  • Strict schemas and guardrails so model output becomes executable actions, not just chat

Challenges we faced

1) Reliable structured outputs (JSON that always executes)

Tool-calling breaks if JSON is inconsistent. What we did: enforced strict schemas, validated outputs, and designed prompts to prioritize executable actions over verbose explanations.

2) Preventing silent errors in automation

When an agent acts on your data, trust matters. What we did: built the editable proposal card as a verification step, so users can correct assumptions before anything is saved.

3) Latency perception and UX

Even short waits can feel like failure. What we did: designed UI flows that show progress and keep the experience responsive, so reasoning time feels intentional instead of “stuck.”

What we learned

  • Prompt engineering is product architecture. The structure of context and tool contracts determines reliability as much as model capability.
  • Trust requires user control. Human-in-the-loop confirmation is the difference between a cool demo and a usable product.
  • State is everything. The agent becomes truly helpful only when it can reason over a persistent loop (inventory → shopping → meal log → workout log), not isolated one-off chats.

What’s next

  • Multi-device support: expand beyond web so users can use the same plan and state across devices, with consistent syncing and a unified account.
  • Expand to mobile after the core loop stabilizes, enabling more natural daily usage (quick logging, on-the-go decisions).
  • Fridge photo recognition (properly evaluated): add robust testing and verification before making it a primary input method.
  • Better joint planning: tighter coupling between meal timing/macros and training guidance.
  • Predictive shopping: anticipate missing staples based on consumption patterns and planned recipes.

Built With

Share this project:

Updates