I enjoy shopping in a well organized and informative shop, where details of the products are clearly displayed. However, sometimes shops ignore the customer experience when displaying their items. And with AR, users can point their phone at the item, and get instant info such as the price, and the description.

What it does

Provide the merchandise backend to scan the items to our database. Provide the customer client side where all the scanned objects can be identified and an information tag with the price and other metadata are overplayed in a virtual work using AR technology.

How I built it

I used ARKit's to build the AR experience. I used Firebase and Google Cloud platform to store the extracted reference model for the items scanned in the merchandise backend. And On the customer clients App, I retrieved the reference model from Firebase in order to identify what's on the user's camera view.

Challenges I ran into

Integration with Firebase: such as serialization of the reference object so it can be stored on a Google Could Instance. Understanding the coordinate system in the AR world, and think about the correct sized of the virtual tags with respect to the object. Version control issues specifically because of cocoa pods usage, it makes it so hard to push and pull codes to collaborate. Getting Wegmans' product info, a sku of "04913207" can't be passed in as an int, since it will ignore the leading 0s.

Accomplishments that I'm proud of

Finishing the minimal viable product. Now, everyone can add scanned objected into my database, and on the client app, it will overlay a informational tag. Thanks to firebase. ARKit tracks object without crashing, and maintain a high frame rate. Metadata such as prices and description are stored along with the model on Firebase, and can be edited in a way that changes will show up instantly on the client side.

What I learned

Be patient and not give up especially when you find out Firebase doesn't allow you to directly download a directory. I was so stressed and about to give up because I thought it would not be possible to use Firebase for storing all my reference models. However, after an hour of research, I finally found a way to bypass this restriction by adding a file list containing all my files.

What's next for Peep

When the merchant backend scans the object to be uploaded, I wish to use computer vision algorithms to auto detect the bar code, so that the user won't need to enter it themself to get the price and name from Wegman's api. I hope to integrate a live Wegmans API integration so the prices can be updated without going through our backend. Also add more details in the overplayed virtual tag such as videos and link to the product page.

Built With

Share this project: