Making healthy and sustainable decisions during grocery shopping is not just a trend, but it's also actually healthy and sustainable! In today's world, people are used to giving in to their hedonistic desires by being able to buy everything their heart desires. Very often, they do so inadvertently due to an abundance of choices with little to differentiate from. Not anymore! In such times of despair, HealthyBoy (the Boston Dynamics robot dog) comes to the rescue. You may have envisioned a future where shopping is entirely online but we are here to buck the trend and make offline shopping hip and fun again!
What it does
HealthyBoy (the robot) aims to provide an engaging end-to-end shopping experience by linking with your shopping profile and helping you select items that suit your lifestyle and are sustainable for the planet. It follows you throughout the store and steals unhealthy and unsustainable food items from your basket when you aren't looking and then runs away. If you still want to have your sinful groceries, you'll have to chase it. In order to ensure that it is actually you, we have equipped it with facial recognition and identification. With its adorable smiley-making skills, it communicates how it feels and what it thinks. Its overall goal is to make your shopping experience more joyful, entertaining and above all: healthy.
How we built it
Lots of screws, yellow metal, and a bit of patience. Jokes aside - there are four main components to this project. Firstly, the robotics controlling in general. Secondly, the facial recognition system, which extracts images from the robot's cameras and matches the faces to your Cumulus accounts. The third part is the person-tracking system, which uses all of Spot's 6 camera-rigs to track its user's movement. Spot fits a human pose to the images and always knows whether or not you are watching over your shopping cart. Whenever you are distracted, it scoops in and looks into the shopping cart with its arm, classifying all items it can see and ranking them using "healthiness" and "sustainability" metrics. It looks for the worst item, steals it right out the basket, and runs away! Lastly, we build an iPhone app that communicates wirelessly with the robot allowing it to show adorable animations and more, making it more alive.
Challenges we ran into
Robotic systems usually involve a plethora of components with intricacies most people may tend to overlook (like our own humble selves). Just connecting all the computers together, reading camera images from the robot's 13 (!) cameras (RGB + Depth), controlling a robotic arm and setting up the development environments used in typical robotics projects takes forever and costs many nerves. Moreover, whenever real hardware is involved, magic smoke appears (-1 battery charger). Take all that and throw it into an action flow and you have yourself a steaming pot of bugs. Yay.
Accomplishments that we're proud of
Building such a large distributed system in such little time was a great challenge. We sort of knew what we were doing but still reached our limits. We needed features the official API for Spot and the Robot Operating System did not support - so we literally had to "hack our way" through this project. We believe we found an awesome solution to accurately grasping the desired item - which is hard if you think about all the uncertainties in our environment. It's about as error-free as programming with goto statements is.
What we learned
Some of us had our first experience with the Robot Operating System (ROS), others deployed object-detection, human-pose estimation, and segmentation models for the first time, but all of us learned that robotic systems are more complex than you initially think. In addition to this, we had the behemoth challenge of integrating all these moving pieces into a seamless end-to-end framework. We would have loved to add more features (such as happily dancing after stealing from you) but ran out of time.
What's next for HealthyBoy
As developers, we are huge proponents of iterative and incremental development and although this is a working prototype, we believe HealthyBoy still needs a few more iterations before it can show up for work at retailers in a safe and entertaining manner. With more time, we would also improve the UI/UX and add more interactive and informative features such as recycling information, automatic checkouts and reward points. We would also like to add more routines to Spot to better suit the diversity of shoppers today. In the future, we will see more robots like HealthyBoy/Spot - and we can't wait to help build this future and get people off their couches and into the stores (hopefully also leading to a barrage of Cumulus and other signups for retailers wink wink). Who let the dog out, eh?