ARMS (Augmented Retail Management System)

• 

  Logic Diagram

• 

  Frontend page

• 

  backend script in action

Inspiration

The pandemic world has pushed to adopt more and more remote shopping experience, creating new challenges and opportunities for many retail and grocery owners. However, majority of warehouse management as well as warehouse stuff are still done by human labor. We aim to reduce the gap between state-of-art remote shopping experience and warehouse/retail automation, by heavily utilizing AMQP protocols, robotics, reinforcement learning as well as mixed reality headsets.

What it does

It allows customers to add grocery items via a Node.js web app, and it will instead send a notification to the retail owner, who can then use mixed reality headset to fetch/manage the items (if available) and put them in the shopping cart for future pickup. The actual warehouse is managed by a robotic arm that could be remotely controlled by human.

How we built it

There are three parts of the project: warehouse (robotic arm), grocery/retail owner as well as the customer. We adopted modular approach and tried out best to decouple all of them, and started tackle the problem separately. We created a web interface for customer to place orders, and used Unity to create a digital twin of the robotic arm as well as a control panel for retail management, which can showcase the number of nearby stores, the amount of information for the grocery items. This is largely achieved by making calls to many NCR API such as their Order and Sites endpoint. We also have designed an experimental full-auto algorithm for automatically picking up the item, though its reliability is largely un-testified.

Challenges we ran into

Accomplishments that we're proud of

We are able to weave so many random things together within such a short amount of time. Further we are able to make successful integrations of NCR's API and use their HMAC authentication despite so many struggles.

What we learned

Plan ahead and do not waste time

What's next for ARMS (Augmented Retail Management System)

The experimental automatic grabbing algorithm that uses reinforcement learning (PPO) is going to be deployed and tested further, adding more automation to the project.

Built With

• c#
• c++
• ncr
• python