We list Thai common manus and make summary of the ingredients. Then take photos each of them on the white plate as the background.
Set up table. Taking photos under a light.
Import the photos to Computer Vision Annotation Tool (CVAT). Segment each ingredients by hands. Give them their labels.
Broccoli after segmentation
The output from CVAT has 2 types. First is real picture. Second is mask picture. The trainset is the information inside segment area.
Put a model on Django web framework. Upload new picture will give a result as text in the bottom of the page.
- Calculating calories by hands is difficult and take a lot of time. Delivering simple process to list ingredients will help people who struggle with health problem has a better way to plan their meal. Building the application receive the meal photos and return a list of ingredients. Users can use that list to calculate the calories further. About our dataset. Thailand is our home, so we present our own ingredients as a dataset. Some local ingredients is not globally well known. We expect application help anyone interest in Thai foods know more about our recipes.
What it does
- User upload the picture of dish to application. The picture will be the input of the model and predict the ingredients in dish. The result is the list of ingredients consist in that dish.
How I built it
- First, we don't have any data. We list the ingredients from common Thai menus, put each of them into dish and take a photos. Computer Vision Annotation Tool (CVAT) is used to segment ingredients from each picture. The output is a mask picture of the ingredients. Each ingredient has different masking color with the same black background. Combine information of real and mask pictures. We send the data of real picture in the area of masking to the model.
The model we use is Mask R-CNN from the "TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL". We use Pytorch to train model and save it. Then load model to Django web framework. At Django, we build the web application with the "choose file" and "upload" button. The upload picture will plug into the model and give a result of ingredients in that dish.
Challenges I ran into
- We try to collect each ingredients pictures similar to the one in real dish as much as possible. Some ingredients can come in many variations. If we do this process well, our model can predict a lot easier because the dataset is cover the distribution of real dish.
- Time limit cause us to label only feel categories. The result in some ingredients is very good. Some are not.
- Some ingredients have same color and almost the same shape. The example is cooked rice, squid. The model still make a wrong decision.
- The integration of model and Django web. There is long loading time predict the result.
Accomplishments that I'm proud of
- In common competition, they always provide the dataset for us but not this one. We start to list the common Thai menus on the internet. Make an ingredients summary list and go to market to buy them. Then taking their photos. This is the first time we create the data by ourselves.
- We aren't familiar with building website. Integrating website with model for the first time make us proud.
What I learned
- The result will be good or bad depend on collecting raw data process. We have to take the picture of each ingredient as same as the one in real dish. This make our data cover the distribution of the real dish. Then the result will be good. Example is shrimp. Shrimp has many variation in Thai dish such as raw shrimp, cooked shrimp, peeled shrimp. We have to take photos of these shrimp variation and label it as one group. Making the application work in real dish.
- How to build website and integrate with model. Especially how to make picture in right format to show on Django frontend.
What's next for
- Next step of this project is to calculate calories of dish. User send dish picture to application and it will return list of ingredients and calories based on approximate volume. This will make the process more simpler than calculate it by hands. People who struggle with health problem can know roughly detail of calories in the dish before eating. Then adding function to recommend the proper dish base on user health.