Not being satisfied with being unable to reproduce culinary experiences had at restaurants or from images of food found on popular image hosting websites (Instagram ;-)), we've created a Facebook Messenger message bot that generates from recipes from the photos that are submitted by our users.
What it does
Facebook messenger chatbot that generates food recipes from food pictures.
A presentation of the idea can be viewed here
How we built it
The messenger application consists of a nodejs server reading in user inputs via a webhook and responding with messages returned from a backend application that is communicated with asynchronously.
For the backend, we created a small Flask app that integrates a Facebook curated PyTorch machine-learning model InverseCooking. We preprocess and feed these images to the model which then generates:
- a recipe name
- set of ingredients
- set of instructions
Challenges we ran into
We struggled greatly with the Python module system which has historically been a pain in the ass.
Accomplishments that we're proud of
What we learned
How to integrate PyTorch with Facebook messenger through the API.
What's next for granmabot
We predict to include additional features, such as:
- Estimation of the nutritional values of the recipes.
- Providing an alternative version of the same recipes to be more inclusive (veganism, food intolerances, etc.). (probably including Food2Vec)