Food2Vec

Let us cook.

Comment

Inspiration

Food recommendation today is largely based on ratings, tags, or surface-level ingredient matching. Inspired by Word2Vec, we wondered: what if food could be embedded the same way words are where similarity emerges naturally from structure and context rather than manual labels? This led us to Food2Vec, a system that encodes food not just by ingredients, but by how it is cooked, transformed, and served, capturing taste and culinary similarity in a vector space.

What it does

Food2Vec creates vector embeddings of food directly from recipes. Instead of treating a recipe as a flat list of ingredients, we model it as an ingredient–action–state graph, where:

  • Ingredients are transformed through actions (e.g. dice, fry, boil)
  • The order of operations matters
  • Each step produces a new state that can be used in later steps
  • Serving style (e.g. taco vs burrito) is also encoded

From this representation, Food2Vec can:

  1. Find similar foods by nearest-neighbor search in vector space
  2. Build a user taste profile based on likes and dislikes
  3. Recommend foods tailored to an individual’s preferences
  4. Match users with similar (or complementary) taste profiles to help groups decide what to eat together

Ultimately, Food2Vec aims to solve both:

  • “What should I eat?”
  • “What should we eat together?”

How we built it

  1. Recipe representation We parse recipe instructions into ingredient–action–state graphs. Each cooking step consumes ingredients or intermediate states, applies an action, and outputs a new state. Every direction can be thought of as a state machine that takes ingredients or ingredient-states as inputs and outputs a new ingredient-state that can be an input for another direction. This effectively converts a recipe into a Directed Acyclic Graph.

Example: tomato + onion --(stir fry)--> stir_fried_tomato_onion

  1. Taste-aware encoding Ingredients are tagged with taste attributes (e.g. salty, sweet, umami), and then encoded using MiniLM to capture semantic similarities between broad classes of ingredients (chicken / poultry)

  2. Vector embedding Recipes are converted into vector representations that encode:

    • Ingredients
    • Cooking processes
    • Order of transformations

  3. Vector database & retrieval We store embeddings in a vector database and perform similarity search to power food discovery, recommendations, and user matching.

  4. User profiling Users swipe through foods, and their likes/dislikes move their position in the same vector space, forming a taste passport.

Challenges we ran into

  • Parsing unstructured recipe text into consistent actions and states
  • Preserving order and transformations, which are often ignored in recipe models
  • Time constraints, balancing modeling ideas within a 24-hour hackathon
  • Deployment and integration issues, especially under CI constraints

Accomplishments that we’re proud of

  • Designing a novel ingredient–action–state graph representation
  • Encoding not just ingredients, but encapsulating the process of cooking and flavours evolving together
  • Creating a shared vector space for both food and users
  • Demonstrating how embeddings can enable individual and group decision-making

What we learned

  • The way data is represented can matter more than the model itself
  • Food is inherently sequential and stateful, making it a great fit for graph-based representations
  • Even simple taste heuristics can meaningfully shape embeddings
  • User preferences can be modeled naturally as movement within an embedding space

What’s next for Food2Vec

  • Improve automatic extraction of cooking actions using LLMs
  • Expand and balance the dataset across more global cuisines
  • Build richer group-matching features to help friends reach food consensus
  • Integrate with restaurants to recommend places based on food similarity rather than ratings alone

Built With

Share this project:

Updates