CHECK IN 3
Introduction: What problem are you trying to solve and why?
Have you ever seen a food picture on social media and wished you could make it yourself? It’s extremely hard for humans to be able to look at a picture of food and devise a recipe to recreate the dish. We are attempting to train a neural network to solve this problem. We will take in inputs of food images and output a step-by-step recipe to recreate the food in the picture. Using this model, we hope to make cooking more accessible for the average home cook, so that their cooking repertoire is not limited to the cookbooks they own, or the well-taught tutorials they can find online, but any picture of food on the Internet.
Challenges
So far, the hardest part has been determining what aspects of the previous implementation in pytorch we should keep and convert to tensorflow, while filtering out parts that are extraneous. In trying to convert some of the classes and functions to tensorflow, there have been decisions that we’ve had to make without being able to verify them in the scope of the entire project. In addition to this, converting some of the pytorch methods to tensorflow has been quite challenging. Not only are there quite a few pytorch methods used throughout the code, but the methods are quite significant, requiring a lot of research on how to convert them to tensorflow. So, this likely means we will have to dedicate more time in the future to putting the different pieces of the project together.
Additionally, in pre-processing, since our dataset is different from the one used in the paper, we needed to obtain the data from a CSV into the correct Python data structures before using similar pre-processing functions as the original paper. This required us to convert it into a pandas dataframe, whereas the original paper used the Python json library to retrieve data from their dataset’s JSON files.
Insights
So far, we haven’t been able to see any results, as we are still in the process of re-implementing the model. We have finished converting the preprocessing and some modules of the model architecture.
Plan
At the moment, we still need to implement the training loop of our model, as well as putting the two components of the model together. Since we have been working on different parts of the model separately, we envision that making sure each part is compatible with each other may take some tweaking once the individual parts are done. We will likely need to dedicate more time towards testing our implementation once it’s done, as there could be bugs that aren’t obvious enough to catch before trying to train our model.
Log in or sign up for Devpost to join the conversation.