Foodies

• 

  Get recommendations and prices of similar items based on customer transaction history

• 

  Take a picture of store products you want recommendations on

• 

  Select the most relevant descriptors of an image

• 

  Choose an item from the current store's catalog using NCR's API

• 

  Get classification and concept tags about an image using Clarifai's API

• 

  Bare-bones launch screen; focused around two buttons

• 

  Tag screen; user selects which clarifai tags represent the image

• 

  Uses android's built in camera UI which lets users confirm or retake photos

• 

  Tag screen; user selects which clarifai tags represent the image

• 

  Android provides a UI for taking just one photo

Inspiration

This app was inspired by recommendation systems that we typically see in online shopping. Our team wanted to use computer vision and AI to bring these same recommendation systems to everyday life.

What it does

Let's say you are out at the grocery store, hardware store, clothing store, etc. Our app allows you to take a picture of an item you are going to purchase or think you might want to purchase. It then helps you identify the object and gives you recommendations of other items you might want to purchase at the current store based on the history of other customers' transactions and what items are typically bought together. It will present you with these options and the cost of the items it is recommending.

How we built it

We used a couple powerful APIs to implement this project. After the user takes a picture of an item, that picture is pushed to the Clarifai API which will return a series of tags of the image describing the likely objects present, and the user can indicate which tags are best. We then collected a catalog of the items that were present at the store using NCR's API platform. We polled the catalog based on the image tags for products that are most similar to the tags, and returned this list of products to the user. The user can then scroll through the list and select the product that they had taken a picture of. Finally, we once again use NCR's API for the store to access the transaction history of products in the store in a certain time period (for example, the past 3 days). Given the item that the user selected, we analyzed the transaction history using some probability to predict which items are most likely to be bought with the item the user selected.

Challenges we ran into

The main challenges we had were figuring out how the APIs worked. Troubleshooting how to get the android UI to with Clarifai running asynchronously Making an http call in java Navigating android studio as a first time user

Accomplishments that we're proud of

We are proud that we were able to make a cool hack that works pretty well. Given just a bit more time, we could launch it as a well-polished android app

What we learned

We learned a lot about different API calls and how to integrate them in our own software. We learned how to use android's API We learned how to make an http request and parse json data file

What's next for Foodies

Some things that could be improved upon are the way that the image tags and store catalog items are correlated. The simple string matching method that we used does not work best in all cases and some form of semantic matching might work well. Furthermore, we'd aim to finish launching the android application. The python app works well for demonstration, but the UI is limited in scope and we could not practically launch a consumer facing application with python.

Built With

• android
• clarifai
• ncr-platform
• opencv
• python