Everyone gets choosy when they have options to eat. Everyone has their own preference but sometimes when life gives you too much free food, making the right choices is of great significant value.

Imagine pointing your camera at your food and getting to know which one is gluten free? Which drink will give me the most caffeine? Every existing app scans barcodes or inventory of products. We are here to do something new. If you see a table filled with food, how do you decide what to eat?

Spot allows you to tap and select all products you’d like to compare and asks you the filters you need to add. If you choose “Egg-Free” SPOT will tell you which of the products that you chose is egg-free - in augmented reality!


One of our team members had a hard time figuring out what he could eat at HackHarvard without triggering his nut allergy. Another is a vegetarian and had to ask about whether the food being served was vegetarian or not. There is no quick an easy way to determine information like whether an item is gluten free, vegetarian/vegan, nut free, etc. That is when we realised that it would be much easier if we had something that could find the details for us so that we wouldn't have to Google each product ourselves.

Problem Statement

Reduce the time spent reading nutrition labels and making Google searches while shopping to a tiny fraction of what they were earlier by making a system that allows one to intuitively and conveniently compare food items based on filters like gluten free, egg free, caffeine content, etc.

How Does It Work?

Spot is an intuitive solution that spells the end for the problem of spending time with nutrition labels and Google searches. Just take out your phone, point your camera at the products you want to compare, and you can see which of the items you’re looking at are gluten-free, contain dairy products, contain caffeine, have sugar, and a lot more.

Technologies & Platforms Used

  • ARKit
  • coreML
  • Swift 4
  • Vision Library
  • Turi ML framework
  • JavaScript
  • REST API using NodeJS
  • MongoDB as the noSQL database
  • Google App Engine for hosting the server

Challenges We Ran Into

Recognising products and then using the models trained by us so that the app doesn't get confused between two similar products was a major challenge, since we had to make our dataset from scratch and didn't have many images. Also, we had to make sure the UI is simple and does not overwhelm the user with the huge amount of information associated with a food item.

What's Next?

A lot!

We wish to extend this app to categorise all kinds of food in retail stores by making use of a bigger data set, more filters and more parameters for comparison. Furthermore, we'd like to have personalised recommendations for users and integration with healthcare platforms to alert users when they look at foods that can harm them and extensions of the application for store managers so that they can categorise their products effectively.

Share this project: