Bay Leaf

Inspiration

In all honesty, cooking is hard. It's a skill that takes years and years to refine, and there's always room for improvement. When cooking, we often find ourselves searching up things like:

What ingredients can I use if I don't have -- ingredient?
How many grams of flour are in a cup?
"Can I eat the pit of an avocado?" -Saif (our teammate)

For new cooks, following a simple recipe can take hours because there are so many things to learn at every step. We want to make the cooking process less daunting.

What it does

BayLeaf is an AI recipe app to simplify the cooking experience. BayLeaf allows you to cook any online recipe with ingredients, nutrition facts, and step-by-step directions at the tips of your fingers!

Before you cook, BayLeaf helps you check your ingredients and keep track of their quantities. Use this to quickly check if you have the necessary ingredients in your fridge.
As you cook, BayLeaf can answer any question using the context of the recipe and relevant images and videos from the internet. You can ask text questions, verbal questions, and questions with pictures.
As you cook, BayLeaf's UI supports hands-free mode to make it easier to follow recipes with dirty hands and make cooking as simple as possible. In hands-free mode, BayLeaf listens for voice commands to navigate the recipe and can answer any verbal questions.

How we built it

BayLeaf's AI is built on 2 different models. We rely on techniques like RAG and fine-tuning to improve the responses of our models.

For general recipe questions and recipe parsing, BayLeaf relies on Claude running on AWS Bedrock to answer questions with multimodal data (text, images, videos) and provide multimodal responses with RAG using the Google Search API.
For ingredient checking, BayLeaf utilizes a fine-tuned instance of Moondream (a lightweight vision language model), this instance was trained on the GPU Max online machines provided by Intel utilizing IPEX. The model was then deployed on a FastAPI server on an Intel Tiber Developer Cloud Compute VM.

BayLeaf's front-end is built on React Native for mobile development with a focus on convenience and hands-free interaction. The back-end uses FastAPI and Redis for recipe caching.

Challenges we ran into

When fine-tuning our Moondream instance, we had difficulty setting up training scripts to be compatible with IPEX. We swapped out various parts of the training pipeline to run on GPU Max.
When using Claude for recipe parsing, the requests often took a long time initially. We used Redis to cache common search queries and recipes to make this faster.
Finding a reliable recipe search source that had the right data we needed, after going through many APIs, we ended up landing on Google's Custom Search API due to its vast amount of data and reliability.

Accomplishments that we're proud of

We tackled React Native for the first time in a complex project. While challenging, we managed to create the core features we had planned.
We're proud we got to explore and use newer technologies like Intel's Developer Cloud and Amazon Bedrock with Claude 3.5, which helped us expand our skills and knowledge.
We're getting a chance to teach everybody how to really cook, whether they're a beginner or professional, this tool can truly be for anybody

Intel Track

What's next for Bay Leaf

We intend to become a startup to bring the joy and ease of cooking to people who are beginning to learn how to cook!
Interested in staying in touch? Join our waitlist!