The introduction of the digital commerce platform in 1982 and the arrival of now major platforms back in 1995 ushered in a new era of e-commerce in a never seen before digital format that opened up limitless possibilities. With the advent of contemporary mobile phones, access to such platforms became easier & adoption skyrocketed. Since then, e-commerce has largely remained rooted with more than a decade since the last revolutionary shift in how customers consume these e-commerce platforms.

We relentlessly innovate to assist enterprises to adopt technologies to augment trust among their customers. The Mixed Reality realm - AR, VR & MR has long been touted as a game-changer and perhaps the evolutionary catalyst that has long eluded the e-commerce industry. In a recent study, 61% of shoppers prefer retailers with an AR experience to those who do not. Also, 72% of shoppers made an impulsive buying decision due to AR-enabled experience.*

What it does

FreshKArt is an e-commerce application for iOS devices that nudges users to perceive to scale products in their environment. AR shopping use cases are segmented across three pillars - Try-On that primarily deals with garments, Try-Out for Auto, home accessories & furniture, and Interact for electronics and layout or assemblies. FreshkARt embodies the last two: Try-Out and Interact. The application does it all.

How we built it

With the team's varying background in technology, we made a decision - one that would ensure inclusivity of all team members regardless of what technology we built the prototype in. Each team member was assigned numerous responsibilities - From sharing a load of our day to day activities to researching the Tchibo brand, principles, and aspirations so the designers and developers could focus on the application.

We zeroed in on The Tchibo coffee machine as our coup de grâce and crafted the model from scratch using 3DS-Max with inspiration from the official website. Smartwatch, furniture & a dress model was included later to further expand the application's horizons into what materials can be visualized in AR using iOS' ARKit including a slide-out menu with the machine that offers information on what coffee beans from Tchibo produce the best aromatic coffee. During our test runs, we were always left with a desire to learn more about what product we were looking at, and the inclusion of a conversational bot from AWS Sumerian seemed relatable to the experience.

Challenges we ran into

Personifying the The Tchibo coffee machine was a hurdle we overcame by resorting to building it on our own. We couldn't procure a royalty-free base model to build atop. So our in-house product designer came up with the mesh & textures by recurrently referring to the official website for inspiration. On the implementation end, we attempted to create a JS interface bridge to execute native functions through JavaScript. This would have helped us for a hands-free conversational experience. Alas, with imminent deadlines, we were unable to. AWS Sumerian requires Javascript to extend functionality. For a team that comprises mobile developers, this became a problem. However, we persevered and were able to include unidirectional prose.

Accomplishments that we're proud of

The hard work that spawned The Tchibo coffee machine 3D model has brought us immense satisfaction. If given an opportunity, we'd like to further refine our crude effort and refine our design to ensure it meets the highest quality standards that Tchibo prides itself on.

Managing daily responsibilities towards our organization & the hackathon astride drove the team into longer working hours while managing time for our families during WFH. Exquisite time management, dissemination of responsibilities, and clear communication have been amongst the best our team has produced in recent times and we're proud of everyone on the team!

What we learned

ARKit enables people to augment products like smartwatches & appliances in their environment. Fusing ARKit with AWS Sumerian for conversational experiences was a first for us.

What's next for FreshkARt

  1. Inclusion of AR-powered remote assistance for technical & installation troubleshooting for customers post-purchase.
  2. Wrist tracking & moving the smartwatch along the path of the wrist is perhaps the next big thing we'd like to try out by leveraging ARKit's 3D body tracking API.
  3. We're proud of our effort towards crafting The Tchibo's coffee model. We'd like to further refine this design and include animated sequences that cover a range of tutorials found on Tchibo's official website.
  4. The AWS Sumerian conversational bot at this time restricts itself to a monologue. We'd like to expand this use case & integrate it with AWS Lex for conversational capabilities and execute animated sequences on the model based on customer queries.

How to try the application

  1. We've included a QR code below.
  2. Use your iOS device and point your camera to the QR code. It should redirect you to an AppCenter page with an 'Install' button. Tap it.
  3. The iOS system should request your permission to install an application. Tap on "Install". It's completely safe! We downloaded it too!
  4. If you run into any issues, The "Downloading..." page has a list of FAQs that should be able to resolve any queries.
  5. Upon successful installation, you need to go to Settings > General > Device Management and trust HCL Technologies Ltd. certificate. Again, safe!
  6. You should be able to run the app henceforth.

FreshkARt QR Code

How to use the application

Note: This is a prototype & is governed by the following.

  1. Open the FreshkARt application and tap on any one of the categories at the top. Alternatively, you may also use voice search by tapping on a microphone button on the top search bar. The voice search accepts a handful of keywords only: Coffee machine, Ladies, fashion Trends, Furniture, Apple Watch, Dress, Bed.
  2. 4 products, 1 from each category may be viewed in AR mode. These are signified by an AR-View image against an article.
  3. On the article detail page, tap on View-in-AR to start an AR experience.
  4. Hold up your camera so the viewfinder may perceive the environment & point towards a surface that has distinct textures (It helps the camera understand the environment). Identifying a surface may take a while. Upon successful identification of a surface, a text should appear that mentions, "Surface Detected".
  5. Tap on the yellow box in the center of the screen to place the AR model.
  6. Click on the sidebar icon on the right edge of the screen (three-dotted icon) for more options.
  7. The sidebar consists of the following options: Scale, texture modifications, View product details in PDF, and Avatar for voice product description.

* Source: Interactions "The impact of Augmented Reality in Retail"

Built With

Share this project: