Everyone is getting a receipt when shopping some groceries. In most of the cases, it is only used to check if there is some mistake or to have some proof how much we spent. However, the full potential of the receipts was never used before. That is, to provide us in a very convenient and easy-to-maintain way of our daily intake of the essential nutrients and the expenses we have to buy that. The app was built during the Open Food Hack at EPFL Lausanne.
What it does
Using the OCR techniques the app is extracting the name and the price of the products listed in the receipt. Then, it queries the Open Food and Open Food Facts data sets in order to get the following information: the original name of the product, total calories, amount of fats, carbs and proteins, as well as the price, the type of the product and the nutritional grade. All of this information is stored and used for visualization and tracking the user's history of bought products.
How we built it
Of course, using the Django framework to manage the entire Web Application. First of all we used the Google Cloud Vision API and OpenCV for the OCR task, i.e. to extract the text from the receipt image. That includes the name of the product and it's price.
Using the extracted name of the product, first we queried the Open Food Data via it's Elastic Search support in order to get the full name of the product, it's barcode and the nutrients. Then, we queried the Open Food Facts data in order to retrieve the type of the product, it's nutritional grade and when possible the geo location of the factory where it is produced. Furthermore, this data is stored in the local user database, such that the user can keep track of all the products information as well as the expenses for buying them.