VCart

A mixed reality remote shopping experience on your cell phone

Comment

Inspiration

Because of Covid 19, shopping in supermarkets has been severely restricted. On the other hand, Mainstream shopping apps such as Instacart still relies on human shoppers to search the items, increasing the chance of exposure to Covid 19. We decided to tackle the hardest challenge during Holiday hack--we aimed to provide a 100% safe, virtual shopping environment that eliminates the need for human shoppers to search around the shop.

What it does

Since this app is built with ARkit and Reality Composer, the user can actually shop in 3D using augmented reality and can use hand gestures to select the items. By grabbing whatever items the user wants to purchase, the items automatically get added into the cart and he or she can proceed to checkout, by tapping the screen. The user can also see what his/her favorite and most purchased item using firebase, as the app tracks how many times the user buys a certain item in the past. The app allows the user to walk along the aisles with shopping cart with them at all time, allowing for a real shopping experience, without the need to go to the actual supermarket. With VCart, mixed reality shopping can be done at home in a much safer way amongst the pandemic.

How we built it

First, the scenes for the supermarket are built using Reality Composer. This makes sure that the correct food items, signs, and shopping cart are being correctly updated as per the users actions. Next, coding in Swift, we used ARKit to add the functionality to the users hand gestures in order to select the respective items. This was done using Apple's vision framework with allows to find the finger joints and use those coordinates to select the respective item. The rest of the functionality for adding the signs and proceeding to checkout was further added.

We also used Google Cloud firebase in our project. Google cloud Firestore stores the amount of time each item is purchased. This data was then used in Reality Composer to display a sign to show which item is the most purchased from the user, so the user can directly interact with the data.

Challenges we ran into

The biggest challenge that we ran into in this project was using the Apple Vision framework Hand Pose library to transform the object's 3D coordinates to the finger 2D coordinates. This was challenging, since the food items were 3D so they could be dragged using depth Data and more advanced sensors. However, since we were restricted to our cell phone camera, we had to think of a way to transform the 3D coordinate into a 2D plane and compare the coordinates accordingly. After many hours of testing, we were able to transform the coordinates and successfully select items and put them in the cart respectively using a transformation matrix. Some other challenges we ran into was using the Reality Composer framework. Many times, when switching scenes, the rendering of the respective images and objects failed to load. We solved this by merging multiple scenes to prevent rendering issues.

Accomplishments that we're proud of

Mixed reality is a new concept, since combing the AR world to the physical world has hardly ever been done. VCart was able to combine these two worlds and that is a major accomplishment. Using the hand coordinates and transforming the 3D object coordinates, so that the object can move to the cart, we can use the physical hand and the augmented object and combined the two to form a synchronous action. This was really difficult, and took many hours to debug and test partially because of the lack of documentation since Apple only announced it since 2020. We think that combining the physical world to the augmented world is the next step for technology and our app was able to merge the two, making a seamless transition for the user to virtually shop in 3D.

What we learned

For some of us, it was the first time making a AR app and the idea of using Realty Composer was fairly new. We learned how to merge scenes and add functionally to 3D objects and calculate the respective coordinates. We also learned how to use Apple Vision library and machine learning framework to collect the hand coordinates which was a new idea. Because we have no access to depth data, we had to conquer sensor limitations by transforming 3D coordinates of the virtual object into 2D and compare with that of finger joints coordinates. We also learned how to merge the objects from Reality Composer to Swift code to add more complex functionality. Moreover, we learned that the Firebase database query functions are asynchronous and we had to take that into account. Most importantly though, we learned how to continually debug and finally reach our goal no matter how many errors we got.

What's next for VCart

Future steps for Cart, would be to use more advanced sensors such as LIDAR. Using these sensors, we can capture depth data and actually move the 3D object with our hand into the shopping cart, to create a more realistic user experience. We also want the vision library to include more hand gestures, so the user can simply use their hand to grab objects, remove them, or perform other actions. We would also like to implement our app with robotics so that the shopping experience can be fully automnous and the robots can shop for the people, instead of humans.

Share this project:

Updates