Nutrifico: Snap, Detect, and Discover Nutrition - 17

• 

  Demo 1

• 

  Demo 2

Inspiration

Tracking nutrition can be complex and often requires extensive manual input. Barcode-based systems fall short when it comes to home-cooked meals or dining out. Our team was inspired by the need for a more accessible and intuitive way to determine nutritional information from meals without relying on barcodes or complex databases. Nutrifico bridges this gap, providing a simple yet powerful solution for anyone seeking to understand their meals' nutritional content using foundational models.

What it does

Nutrifico allows users to upload a photo of their meal, and our AI-powered system automatically detects the dish, predicts its recipe, and generates comprehensive nutrition facts. It simplifies the process of understanding what's on your plate and provides interactive tools to customize recipes or seek nutritional advice through a chat feature. This approach is designed to make nutrition tracking easier and more engaging.

How we built it

Nutrifico was built with Next.js, using its Server Components to create a robust web app that efficiently handles server-side processing. We use foundational models hosted on Replicate to convert dish images into text, which is then analyzed by GPT APIs to predict recipes and generate nutrition information. To maintain high accuracy, we employ a vector database with over 300K food items and their associated nutrition facts sourced from the FDA. This database allows us to cross-check GPT-generated results and ensure reliable nutritional details for each detected dish. This integrated approach balances scalability with the precision required for accurate nutritional tracking.

Challenges we ran into

One of our biggest challenges was achieving accurate content prediction from image inputs. The inherent variability in dish presentation made it difficult for our model to consistently identify dishes. Additionally, the GPT hallucination issue posed a significant obstacle, as it sometimes led to inaccurate or nonsensical results, especially in generating nutrition facts. This required us to implement additional validation steps and cross-checks with a robust vector database to ensure the information presented to users was accurate and reliable. Despite these challenges, we continued to iterate on our approach to improve the precision and reliability of our predictions.

Accomplishments that we're proud of

We are proud of successfully building a system that can accurately detect a wide range of dishes and provide detailed nutritional information. Our conversational AI feature allows users to engage with the app in a meaningful way, offering recipe suggestions and personalized advice. The project's intuitive design and user-friendly interface received positive feedback during testing, reinforcing our belief that we are on the right track.

What we learned

We gained valuable insights into image captioning and the generation of nutrition facts. Understanding the variability in food imagery was crucial, as was addressing GPT's tendency for hallucination. Through this project, we learned the importance of robust training datasets and reliable data validation to ensure accurate results.

What's next for Nutrifico: Snap, Detect, and Discover Nutrition

In the future, we plan to expand Nutrifico's capabilities by incorporating additional dish types and extending the chatbot's functionality. We aim to add more interactive features, such as personalized meal plans and health-related advice. We also plan to explore partnerships with nutritionists and dietitians to ensure the app provides accurate and valuable information. Additionally, we are considering expanding the platform to include mobile applications for greater accessibility.

Built With

• nextjs
• openai
• pinecone
• python
• replicate