IdentifiAR

Shop smartly. With IdentifiAR, make calculated purchases instantly with product research at your fingertips.

Comment

Inspiration

Inspired by Amazon’s push towards Amazon Go and the “grocery store of the future,” we took our passion for augmented reality and built the experience we've always dreamed of living.

What It Does

Revolutionizes the “point of sale” aspect of shopping by giving the consumer all the relevant information they need to make an informed purchase, including alternative choices to drive pro-consumer competition. As over 60% of buying decisions are made at the point of sale, it is critical that we provide as much insight to the customer as possible during this time period.

Analyzes the selected item, and returns information about said item, including 3D models of our alternative suggestions. If time permits, a buy button will add aforementioned suggestions to a shopping list on our android app.

How We Built It

We knew we wanted to work with the HoloLens for its strong AR capabilities, with the Microsoft Azure cognitive computer API for its computer vision properties, and with the NCR API for the extensive transactional dataset. To achieve this, we split up our team, with Vishakha and Josh concentrating on the backend and Sunwoo focusing on the front-end AR display. In doing so, we were able to split up the learning of specific APIs very early on, allowing strong concentration.

We first constructed the backed in python for the quick scripting and layout, and then refactored the python backend into C# to fit the Unity environment. Our front end development was primarily focused around making the user experience as seamless as possible, and so a significant amount of time was spent configuring everything to run smoothly on the HoloLens.

Challenges We Ran Into

The NCR API was tricky to work with, as many API calls had required params that could only be gotten through another API calls, and thus we were stuck for a little while, chaining calls together before we figured out a workaround. However, the NCR team was very helpful and provided invaluable insight into how their API calls should be constructed.

In addition, we ran into an issue an hour before submission that proved to be very difficult to work through, as we were all running on no sleep, which hampered our problem solving abilities. Thankfully, we were able to rally together and crack the bug.

Accomplishments That We're Proud Of

As a team, we’re very happy with our end results - we took a platform none of us had worked in before, wrote a backend for the HoloLens platform in a language we had no experience with (the backend team hadn’t touched C# before), and successfully integrated these working components while navigating new APIs and overcoming the logistical hurdles of getting it done during a 36 hour hackathon.

In addition, we created a project that walks amongst the future of retail, and that’s something we’re very excited about.

What We Learned

Collectively, we learned how JSON is packaged, how to program in C#, a bit about how computer vision works with Microsoft’s cognitive services, and a lot about the HoloLens platform. We also brainstormed a lot of other projects we could work on with these technologies in the future.

HackGT was a massively educational experience, and we’re very glad we came!

What's Next for IdentifiAR?

We want to make the item recognition more robust, as currently we have to have a specific background for a successful query to go through. We’d also like to explore adding additional details about the products selected, based on feedback from users. Star ratings and company history would be useful to know at the point of sale, and will definitely be included in IndentifiAR 2.0.

Share this project:

Updates