Not being satisfied with being unable to reproduce culinary experiences had at restaurants or from images of food found on popular image hosting websites (Instagram ;-)), we've created a Facebook Messenger message bot that generates from recipes from the photos that are submitted by our users.

What it does

Facebook messenger chatbot that generates food recipes from food pictures.

A presentation of the idea can be viewed here

How we built it

The messenger application consists of a nodejs server reading in user inputs via a webhook and responding with messages returned from a backend application that is communicated with asynchronously.

For the backend, we created a small Flask app that integrates a Facebook curated PyTorch machine-learning model InverseCooking. We preprocess and feed these images to the model which then generates:

  • a recipe name
  • set of ingredients
  • set of instructions

Challenges we ran into

We struggled greatly with the Python module system which has historically been a pain in the ass.

Accomplishments that we're proud of


What we learned

How to integrate PyTorch with Facebook messenger through the API.

What's next for granmabot

We predict to include additional features, such as:

  • Estimation of the nutritional values of the recipes.
  • Providing an alternative version of the same recipes to be more inclusive (veganism, food intolerances, etc.). (probably including Food2Vec)

Built With

Share this project: