Checkout

• 

  Logo

• 

  List view to add new items

• 

  Directions view to find new item

• 

  Checking out an item

Inspiration

Today, traditional stores can not compete with internet giants like Amazon. This seems like a waste, as traditional retail counts for over $4 trillion [1] of the US economy every year. However, the sheer convenience and abundance of choice given to someone when shopping online is slowly driving these stores to extinction - shopping in a store requires a significant amount of time, energy, and the customer _ still _ isn't sure if they have gotten the best deal! However, there is still a large benefit to a walk-in store that online retail has yet to emulate - speed. Even 2-day shipping is often too long to wait for many items, and of course groceries are still off the table in most areas of the world. Our project aims to make the process of walking into a store more appealing to the customer, and reducing some of the pain points illustrated above with some innovative technology including object detection and some great APIs given by NCR.

What it does

Our project is an iOS application that allows customers to get in and out of a store as quickly and painlessly as possible. First, we use NCR's Transactional Data API in order to create a Catalog of items within a given "store". We also use NCR's Item Availability API as a check before showing items to the customer, and if items are about to go out of stock we can prioritize showing them. The customer can create a "grocery list" of items they need, and while they are typing we use the grouping ability in NCR's Catalog API to show items similar to what they are looking for. Once they have created a list of items they need, they can then click on any item once they are in the store, and they are shown a map that tells them exactly where in the store layout that the item is.

The best part is that once the customer finds the item, they can immediately check it out (after which we create an order using NCR's Silver API) by simply holding their phone camera over it. We use a MobileNets object detection model in order to determine what item they are holding, and once we verify that it is the item they wanted, we create an order. We decided to use object detection because (1) it is much more convenient than searching for a barcode, and the purpose of the application is to make shopping in a store more convenient to compete with online retail, and (2) because this method can extend to shopping for groceries or other items that are freeform, and don't usually have barcodes attached.

After the customer has found and checked out all of their items, they can simply walk out of the store! There is no reason to wait in a line, or ask an employee for help finding an item. In this way, we make the shopping experience as simple as possible.

How I built it

We used a MobileNet model trained on ImageNet found open source, and tuned it to fit objects that we needed to detect. Of course, we also used many of NCR's great customer-facing APIs for tracking our own inventory and what we would allow the user to do. Finally, we used Python to communicate with these APIs, read and write to the catalog, and work with the MobileNet model.

Challenges I ran into

The most difficult part of the project was to get used to the APIs that were given by NCR. Throughout the hackathon, we realized that NCR's APIs are extremely thorough and do their job very well, but as a result of being able to do so much with them, there was a learning curve to figuring out how to do anything with them. However, once we figured out how to work with one, the others came very quickly after. Additionally, it was a challenge working with the MobileNets model and figuring out how it worked, as although we did find it online there wasn't very much documentation on how to work with it.

Accomplishments that I'm proud of

That we created a working product that really does have a real-life use case, and that we figured out how to use many new technologies in such a short time.

What's next for Checkout

There's still a lot more that we can do with NCR's APIs that we simply did not have time to explore. For example, tracking users and what they are buying to give them better suggestions as they are making their lists would be a very good feature to have, and one that fits in nicely with NCR's infrastructure. Connecting with payment information held by NCR would also make the process complete (right now, we are faking payments). I could also imagine that we did not completely utilize NCR's discounting tools - I could imagine that being able to see all of the current promotions or discounts going on in the store and nearby stores would be really useful.

Built With

• coreml
• python
• rest
• swift
• tensorflow