AI Fridge Chef

• 

  Detect food & beverages in image (Sonar as vision model)

• 

  Gen recipes for foods, dietary restrictions, and medical diagnoses (Sonar Pro LLM)

• 

  Each recipe has tailored notes and inspiration sources (Sonar Pro search and synthesis)

• 

  Recipes downloaded as a recipe book

AI Fridge Chef is an AI dietary health agent powered by Perplexity's Sonar Pro and Sonar LLMs that scans image(s) of what food you have on hand and based on your dietary restrictions and medical diagnoses, generates breakfast, lunch, and dinner recipes for you using your available food items.

Each recipe is generated using Sonar Pro's real-time Internet search capability to get guidance and inspiration from the latest reputable dietary and culinary sources, synthesized using its advanced LLM engine. Included with each recipe there's a breakdown of the macronutrients in it and notes on how it meets your dietary needs. Also the user can enter more foods not shown or detected in their photos to use in their recipes and each set of generated recipes can be downloaded as a recipe book of healthy and delicious personalized meals.

For scanning the image(s) the user uploads to the agent of what's in their fridge, freezer, cupboard, etc., Sonar is used for object detection (image to text). Then the list of food and beverage items are passed to Sonar Pro to generate recipes tailored for the user's selected dietary restrictions and medical diagnoses.

Check out the prompt at https://github.com/williamcheung/aifridgechef/blob/master/prompts/get_recipes_from_foods.prompt.txt to see how I crafted it to follow best practices provided by Perplexity:

• https://docs.perplexity.ai/guides/prompt-guide#best-practices-for-prompting-web-search-models and
• https://docs.perplexity.ai/guides/prompt-guide#web-search-model-pitfalls-to-avoid

Note the prompt leverages Sonar Pro's ability to return the sources it used for its response, which are included in each recipe as the inspirations for it, after ingredients and instructions (so triple i).

Also, structured output is specified in the prompt itself so that Sonar Pro returns its response in a pre-defined JSON format: https://github.com/williamcheung/aifridgechef/blob/master/prompts/get_recipes_from_foods.prompt.txt#L18

This works well, allowing me to use a Jinja 2 template to render the JSON to recipes in HTML format: https://github.com/williamcheung/aifridgechef/blob/master/templates/recipes.html

To optimize performance, API requests to the model to generate the 3 types of recipes are sent asynchronously, not sequentially: https://github.com/williamcheung/aifridgechef/blob/master/gradio_ui.py#L66

And for handling API rate limit errors, the Tenacity library is used to retry seamlessly with exponential backoff: https://github.com/williamcheung/aifridgechef/blob/master/llm.py#L72

The 3 types of recipes (breakfast, lunch, and dinner) are combined using another HTML template if the user chooses to download their recipes as a recipe book. Each recipe book represents the food images, dietary restrictions, and medical diagnoses the user entered, but the user can change their entries or selections to generate different recipes. As mentioned, they can also type in food items or spices and condiments that aren't in the images they uploaded, and update their recipes with those additions.

The future vision for this agent is to embed it into the control panel of your fridge to auto-generate recipes based on the food in it and other food storage locations, according to your dietary health needs as evaluated by your dietician, and transmit the recipes to a robot chef which will prepare and cook the meals for you on schedule or on demand. Bon Appétit! 🍽️

Built With

• beautiful-soup
• gradio
• jinja
• langchain
• perplexity
• python-3.12
• sonar
• sonar-pro
• tenacity