FoodFit

"Getting fit? FoodFit finds food that truly fits your fitness needs, right from your local supermarket shelves. But we also believe in fitting sustainability into your shop.

Comment

Inspiration

The inspiration for FoodFit came from two common frustrations: the difficulty of quickly finding grocery items that match specific fitness and dietary goals, and the growing problem of food waste. We noticed how much time is spent deciphering labels in supermarkets, trying to align products with nutritional needs (like high protein or low sugar). Simultaneously, we were aware of the significant amount of perfectly good food that goes to waste simply because it's nearing its expiration date. We envisioned an intelligent assistant that could bridge this gap by helping users effortlessly find food that fits their fitness journey while also making a positive environmental impact by prioritizing items that need to be sold soon.

What it does

FoodFit is one part of a three-agent system designed to create holistic, sustainable weekly fitness and nutrition plans.

  • Fitness Planner Agent: Collaborates with the user to create personalized weekly workout schedules tailored to their fitness goals (muscle gain, endurance, weight loss, etc.).

  • Grocery Shopper Agent: This agent takes the user's dietary needs (derived from the fitness plan or stated directly, e.g., "high protein," "keto") and consults its Knowledge Base to understand the nutritional aspects of the required diet and identify relevant product types or ingredients, and it searches a specific supermarket's (e.g., Mercadona) current product availability based on insights from the Knowledge Base and the fitness plan's needs. The key feature is that prioritizes near-expiration Items aiming to reduce food waste.

Working together, the two agents deliver a comprehensive weekly plan to the user, including:

  • A detailed workout schedule from the Fitness Planner.

  • A corresponding, optimized grocery list from FoodFit, designed to meet the nutritional demands of the fitness plan while also considering sustainability by potentially highlighting near-expiration items.

  • Scheduler: This agent takes both the information from Fitness Planner and Grocery Shopper and tries to find a middleground between both, balancing the nutrition, sustainability grocery products and also the exercises

Essentially, the system aims to seamlessly integrate fitness planning and smart, sustainable grocery shopping.

How we built it

We built FoodFit on a serverless, event-driven architecture using AWS, allowing us to build a complex system quickly. These are the components (which are also outlined in the architecture diagram)

  • Frontend: A Streamlit application provides the simple, interactive UI. This is deployed as a container on AWS Fargate behind an Application Load Balancer (ALB) for public access.

  • Backend Orchestrator: The frontend communicates with an Orchestrator service (also on Fargate) which serves as the secure entry point to our agentic system. Additionally, the orchestrator has three services (food, exercises and scheduler) which are Python FastAPIs that are the ones that invoke programmatically the agents.

  • Agent Intelligence (The Core): The entire 3-agent system (Scheduler, FitnessTrainer, GroceryShopper) are built, managed, and configured using Amazon Bedrock. This allowed us to define clear agent roles, prompts, and crucially, integrate their custom "tools."

Data & Tools (RAG + Functions): The agents' tools are the key to their power:

  • FitnessTrainer Tool: We used an exercise dataset from Kaggle. It is stored in an S3 bucket, which the FitnessTrainer agent queries (as a RAG tool) to build its plans.

GroceryShopper Tools: This agent uses a powerful hybrid approach:

  • Bedrock Knowledge Base (RAG): We populated a Knowledge Base with static data from DataMarket (product info, nutrition) for fast, accurate retrievals on general food questions.

  • AWS Lambda Function (Live Data): To get real-time data, the Bedrock agent invokes a Lambda function. This function calls the supermarket's live API (Mercadona) to find current prices and near-expiration items, feeding this live context directly into the agent's decision-making.

  • Data Pipeline: The entire data backend is automated. AWS Glue jobs perform ETL on both static (Kaggle, DataMarket) and dynamic (supermarket API) data, storing the results in S3 for our agents and Knowledge Base to consume.

  • Scheduler: It receives communications from both the other two agents which combines and reasons about a balance that could be sustainable both for the user's health and the world.

Challenges we ran into

  • Agent Orchestration: Getting the agents to collaborate effectively. It took significant effort to ensure the Scheduler correctly passed information (like calorie goals) to the GroceryShopper, and for the GroceryShopper to learn when to use its Knowledge Base (for general facts) vs. its Lambda function (for live prices).

  • Hybrid RAG + Function-Calling: Implementing the GroceryShopper's hybrid logic was complex. The agent had to be smart enough to "self-correct" if its static Knowledge Base data (e.g., "Product X is available") was contradicted by the live Lambda function data (e.g., "Product X is out of stock").

  • Data Integration: Integrating multiple, disparate data sources was our biggest hurdle. We had to write Glue ETL jobs to standardize data from Kaggle (exercises) and DataMarket (products), and then build a separate Lambda function to handle the unstructured, real-time JSON response from the live supermarket API. Especially, the lambda function call gave a lot of problems.

Accomplishments that we're proud of

  • Ideation: We think we have come up with a great idea that with the current agentic-systems can be solved. Additionally, we actually built a live pipeline to fetch actual near-expiration data from a supermarket, making the sustainability goal a core, functional part of the system, not just an idea.

  • Orchestration: We are proud of some design patterns we have followed. For instance, we have used services (trough ECS) to abstract the agents, creating APIs and an architecture that could easily scale with more replicas. We also had to learn to deal with security and networking issues."

  • The Hybrid Agent: Our GroceryShopper isn't just a RAG bot. It successfully combines static knowledge (from its Bedrock KB) with live, real-time data (from the Lambda function) in a single turn. This allows it to answer "What is protein?" and "How much is the high-protein yogurt right now?"

What we learned

  • AWS AI Stack: While having working a little bit in the past, we had to learn about the different services. From the Fargate frontend to the Bedrock agents and Lambda/Glue data pipelines. It has been a great learning.

  • Automation: As we iterated over the code, we found things that could be even further automated. For instance, in the beginning we used Python notebooks for processing the data and loading it to S3. However, as we researched we found AWS Glue being a powerful tool to achieve this with also orchestration capabilities and easy integration and we decided to also include it.

What's next for FoodFit

  • Expand Supermarkets: Integrate more supermarket APIs (Carrefour, Lidl, etc.) to give users choice and find the best (and most sustainable) deals across all stores.

  • Personalization & Feedback: Add a user database (e.g., using DynamoDB) to store user preferences, allergies, and past plans. The agents could then learn and improve their recommendations over time.

  • Cook Agent: Have another agent that collaborates with the rest by taking into account the groceries and the potential meals that can be prepared. This Agent could give step by step recipes to be followed by the users.

Share this project:

Updates