Leaf It to Me

• 

  Main page 2

• 

  Swaps

• 

  Main page

• 

  Ingredients list with swaps

• 

  Flavor map explaining swaps

Inspiration

I actually found the model first. Epicure is an open source model with about 1,800 ingredients that relate to each other from over 4 million recipes. As someone who's passionate about cooking and playing around with different flavor profiles, I thought this was incredibly cool. I thought a really good application of it would be food sustainability: helping people make more intentional choices with their food to lower their carbon footprint and reliance on animal products. It seemed like it would be amazing for food substitutions that stay faithful to flavor profiles and for figuring out which flavors are already frequently paired together (already work well together). It's also not just made those who want to lower their carbon footprint. A lot of people have dietary restrictions that force them to work around typical recipes, and this is great for that too.

What it does

You give it a recipe and pick a diet (pescetarian, vegetarian or vegan). It gives you back the same dish with plant-based swaps that stay true to its cuisine, so butter chicken gets chickpeas and tacos get pinto beans. It shows how much CO2 you save, catches hidden animal ingredients like the rennet in parmesan, and flags nutrients to watch (that you might be losing out on transitioning from animal products), like B12 and iron.

How I built it

I used Gemini to read recipe text and photos into ingredients and weights. Then the Epicure embedding models from Hugging Face to rank swaps by how well they fit the dish and its cuisine, and Our World in Data's food footprint data for CO2, land and water. A hand-built rule list with 54 tests decides what fits each diet, and it's all written in Python.

Challenges I ran into

There were some issues with strictly relying on the model (even the Core model which combines chemical similarity data as well as recipe cofrequency data) to swap out some ingredients that may appear in similar recipes but have better substitutions. The model's closest match to chicken is pork, because they show up in the same recipes, not because one can replace the other. So instead of taking the closest match for some of these ingredients, I use a list of plant-based options for the one that fits the dish best.

Accomplishments that I'm proud of

The swaps actually respect each cuisine -> Indian cuisine dishes get lentils, Mexican cuisine dishes get beans, and Chinese dishes get tofu. I also thought it was really cool to

What's next for Leaf It to Me
Live snapping a photo of your pantry so the app suggests swaps you already have, a mobile app, a web app with a camera button (connecting with phone camera), before/after charts, filling in footprint data for butter, cream and oils, and testing the swaps with real cooks.

Built With

• embeddings
• epicure
• gemini
• google-gemini-api
• hugging-face
• numpy
• our-world-in-data
• pytest
• python
• rapidfuzz
• safetensors