We've all had that situation, where you see a great dish online, but have no idea how someone actually made it. Sometimes, we don't even know what the dish is called, or what ingredients will be needed. This knowledge of what is inside of our food is both important for future cooks, but also for human health, as we should know what is roughly in our dishes before consuming this. This is what Pyra, a new Deep Learning based bot, can help.

What it does

Pyra is Deep Learning, Computer Vision Based Bot, that runs on a Multi-Class Classification problem to identity ingredients and the food type of a given input image. All the user needs to do is input an image of food(Sample Images Provided), and receive back it's name and the 15 most correlated ingredients with it.

How we built it

I built Pyra using PyTorch and PyTorch Image Models to perform transfer learning on the Recipes 5k dataset. The web application was built using React and Flask.

Challenges we ran into

I ran into a large issue of overfitting due to how the Recipes5k dataset was tiny and had only 45 images per class, which isn't nearly enough. To combat this, I learned many techniques of regularization and heavy data augmentation to reduce the amount the model overfit.

Accomplishments that we're proud of

I learned a lot about server hosting and deploying PyTorch models. This was the first model I ever deployed with a React app, and I learned a lot about JavaScript and web hosting, which is something I am very proud of.

What we learned

I learned a lot about web hosting and JavaScript, alongside React Native Web development. Furthermore, I learned a lot about data preprocessing and augmentation techniques to reduce overfitting, due to how there was only 5000 images in the entire dataset.

What's next for Pyra: Ingredient and Food Type Classification.

I plan to gain access to the Recipes1M+ dataset(private only to requests - detailed process) and utilize this to extend Pyra's capabilities to generate recipes, and not only predict the food type.

Share this project: