PantryPal

Ingredients in Demo
Finished Dish in Demo
How It Works

Inspiration

As chefs ourselves, the PantryPal team understands the challenges people face: food waste, inability to decide what to cook, and lack of confidence in the kitchen. Because of these issues, people regularly buy groceries without a clear meal plan or forget about them.

The fact is, cooking feels intimidating and people need assistance. Additionally, recipe platforms make users search through countless recipes that require users to purchase new ingredients. PantryPal asks what ingredients users already have and makes full use of it, and checks in with them after every step to encourage users to make better use of food and give them suitable recipes to cook efficiently.

Overview

PantryPal is an AI-powered web app developed to assist people with cooking based on the ingredients they already have in their kitchen and based on personal factors. Users do not need to search hours for recipes or shop for groceries. Instead, users upload one to three photos of what they have, and AI identifies those ingredients. Next, the app generates personalized recipes based on a variety of factors that we understand are very important to our users, highlighted in key features.

Therefore, PantryPal is not another recipe platform – cooking becomes easier, more accessible, and educational.

How We Built It

PantryPal is a Next.js app on Vercel utilizing Claude over Anthropic SDK. When photos are uploaded, the browser scales it down and sends it to Claude's bot, which then performs ingredient detection, approximation of freshness/quantity, and returning of box coordinates for thumbnails. Every Claude response has to match a Zod schema, so the app gets clean data instead of text it has to parse. To speed up generation, Haiku 5.5 picks three dishes and then Sonnet 5.5 writes their lessons simultaneously so generation takes 23s instead of 40s with retry on validation failure.

Challenges

The main two challenges we faced were integrating the various AI features together into an app and creating an appealing website design for the app. The ingredient recognition, recipe generation, cooking feedback, and other features required different types of reasoning to work cooperatively. We also had to make sure the website UI was user-friendly and simple to use.

Key Features

Ingredient Recognition: Identify ingredients from photos, highlights them, and approximates the quantity Personalized Recipe Generation: Creates three recipes based on available time, number of servings, dietary requirements, and food preferences. Follow-along Cooking Lessons: Provides instructions and explanations of cooking techniques including timers and quizzes Meal Evaluation: Examines pictures of finished dishes to give a score and improvement suggestions

What’s Next?

PantryPal works, but it can be improved. We hope to further integrate image generation models to replace placeholders with related cooking visuals and improve our detection models. Additionally, we would like to develop extra personalized recipes based on previous cooking experiences and culture. Ultimately, we want PantryPal to be the next step in recipe platforms: helping more people cook confidently, waste less food, and become educated in different cuisines.