Let Them Cook

Nearly 40% of food in the U.S. is wasted. Reduce yours by Leting Them Cook, a recipe AI tool that generates fridge-cleaning recipes that are both beneficial to your wallet and the world.

Comment

Inspiration

Nearly 40% of food in the U.S. is wasted. Reduce yours using Let them Cook, a recipe AI that generates fridge-cleaning recipes based on your fridge. Beneficial for both your wallet and the world.

What it does (Frontend)

This app's main function is generating recipes based on the ingredients you already have in your fridge, which is done in 3 steps. First, the AI needs to see what is in the user's fridge. The app opens up the user's mobile camera, the user pans it to the fridge, and the AI scans this picture for ingredients. Second step is the review step; the AI displays what it detected, and the user can add or delete ingredients as they please. (Some ingredients may not be detected due to various reasons such as lighting, angle, whether it's covered in a layer of plastic, hidden away behind a different ingredient, etc.) This feedback will also improve future accuracies as the AI will learn which ingredients it detected and did not detect successfully. Lastly, with this accurate list of ingredients the user is trying to use, the AI generates recipes that uses up as much of this list as possible. Each recipe has high-level information including name of the dish, picture, and description. The user can search to give certain queries and narrow down results. To learn more about a recipe, the user can click on it and a new screen will show ingredients, instructions, and other details given directly from the recipe website.

Other than this main recipe generation feature, we built additional features such as: recent recipes to show up in the home tab for easy access, and a tab to show the entire recipe database to allow queries beyond what the fridge contains, and more.

How we built it (Backend)

In general, AI/LLMs is used in 3 aspects 1) scan fridge 2) generate recipes 3) AI can generate an appetizing photo and description for it.

At the first step of scanning the fridge, the user's image is sent to BLIP-2 (a pre-training method developed by Hugging Face that allows LLMs to understand images. It takes in image and optional prompt, and this take the image is that of our fridge and the prompt is what ingredients there are. Our prompt took a bit of tuning, experimenting what question structure & key words would output the most accurate complete list of ingredients.

For the second step of reviewing ingredients, users can add and delete ingredients and API calls will be made to make these changes to the list. The app then generates recipes by taking this updated list of ingredients and comparing this to each list of ingredients in each row of our recipe dataset. This comparison will identify how many fridge ingredients are used for each recipe, and will take the top 10 recipes that have the most matched ingredients (aka will use as much as possible of our fridge).

Now with the recipe names of these top 10 recipes, we use DALL-E to generate an image of the dish and GPT-4 to generate a description of the dish. These 3 pieces of information is what the user sees when browsing through recipes.

To enhance the user experience and allow for more personalized recipe searches, our platform incorporates a search bar where users can enter custom prompts. By including additional features like 'vegetarian' or 'easy to cook,' users can narrow down their recipe preferences further. We seamlessly integrate Pinecone to upload and persist the recipe dataset, enabling efficient embedding storage. Leveraging the power of Langchain's similarity_search API, we perform a comparison between the user's input and the vector database. This process empowers us to retrieve the top four recipes that exhibit the highest similarity scores, tailored to the user's specific requirements. All of these will help the user decide on a recipe and when they ultimately do and click on one of them, the app will take them to a new window that shows more details of the recipe taken from the dataset and the cooking website itself this dataset took from.

Challenges we ran into

  • Fridge Image Detection has been the most challenging aspect of our project, and we invested a significant amount of time in solving this problem. Initially, we tried pretrained YOLO and image-to-text models. However, YOLO had limited categories, and the image-to-text model could only provide a generic response like "It's a fridge full of stuffs," which was not helpful.

We received some suggestions and decided to explore "BLIP2" and "Image segmentation model+CLIP for object classification." BLIP2 proved to be highly useful as we could improve its performance by refining the prompt. It was also user-friendly, allowing us to test various prompt chains to achieve optimal results. Eventually, we settled on a prompt chain that worked best: "What categories are there in the fridge?" We then processed each category individually.

For some categories, engaging in contextual conversation yielded reasonably accurate answers. However, for others, we had to resort to asking yes or no questions. Also there are some categories didn't provide meaningful information through question-answering. We conducted tests for each category, identifying the most effective prompt for that particular category. Finally, we merged the answers generated for each category. While it may not be 100% accurate, the answers produced through this approach are meaningful to a significant extent.

  • Lack of computational power (GPU), especially for computer vision/object detection portion. We specifically took some time to get BLIP-2 to work, but it ultimately worked on Google Colab Pro with GPU enabled (basically highest possible computation).

  • Lack of time. We had a few features that could've made this app from good to great, but will take a longer time to implement. We also needed more time to experiment with other methods that can detect ingredients more accurately; e.g. we expected image segmenting (Segment Anything) + classifer (CLIP) to work better than BLIP-2 but only pursued this halfway before abandoning it for time constraints.

  • We also encountered some problems when dealing with langchain, pinecone, vector search related things, but finally solved!

Accomplishments that we're proud of

  • We tried a LOT of different LLMs which expanded our knowledge but also give us confidence in our final product as we know we attempted different options and chose the best (but still feasible) one.
  • Our team was also good at splitting up tasks according to each person's strengths while still allowing learning beyond our comfort zone (e.g. I did the design but still got to learn and implement BLIP-2 part of backend)

What we learned

  • [Sandya] Learnt so much in terms of computer vision/object detection from the experienced mentors. Definitely blown away from how many softwares are now out there that make building much easier but I never knew about.
  • [Demi] Within a short timeframe, I ventured into the realm of LLM-related models, delving into tasks like image detection and vector search, which were entirely new to me. Along the way, I encountered various obstacles, but I proactively sought assistance from my mentor and teammates. Through their guidance and support, I was able to overcome these challenges, leading to successful completion of the tasks. Moreover, this journey of exploration and problem-solving has enriched my knowledge and skills significantly.

What's next for Let them Cook

  • Shopping tab: If missing 1 or few ingredients, we hope to be able to webscrape from store websites as well as user location to these stores, and from there generate an insight that tells users 'I just need carrots for this recipe, seems like it's available at a Target 2 minutes away based on a staff report today' (this feature is available on Google right now too). This will prevent users from rejecting recipes too easily and thus give them more options because any missing ingredients are easy to find.
  • Including numbers of each ingredient. We tried to pursue this by experimenting with Meta's Segment Anything + OpenAI's Clip or OWL-VIT in order to identify and box each objects in the picture, then identify what is in each of these boxes -- a method that works as there can be 3 boxes of bananas which can tell the app there are 3 bananas.
  • Allowing number of ingredients to change over time every time the user completes a recipe, which in effect also reduces the number of times user need to scan fridge. This is helpful as it further ensures no food is wasted. For example, app scan fridge on Monday and see 6 garlics -> user complete a recipe that used 2 garlics -> when cooking on Tuesday, app will know there's only 4 garlics left; and do the same for other ingredients.
  • Ability to scan receipts such as those from Costco, which will make getting information such as units, weights, and buy date much easier. By knowing things such as units and weights, the app can not recommend recipes that need more than what the user has; and by knowing the buy date, it will also allow app to infer what is the expiration date.
  • Integration with calendar; in relation to the above, knowing buy date will allow app to know which ingredients are going to go bad the soonest, and that can lead to more affirmative action such as notifications showing up on one's calendar that tell them to eat their apple for breakfast.

Built With

Share this project:

Updates