Inspiration
The Covid-19 pandemic brings to light an increasing need for public hygiene and safety measures to control the spread of dangerous diseases. We present a Touchless Self-Checkout Kiosk -- a technology that allows users to perform hand gestures to operate a self checkout kiosk, so that they can touchlessly scan, process, and purchase essentials while minimizing their risk of contracting disease. Other forms of contactless shopping exist, but they often are only available to customers who have smartphones, internet plans, etc. Our solution is unique in that it has little barrier to entry for the user -- anyone can walk in a store and use the Touchless Self-Checkout Kiosk. Thus, we believe that our solution will greatly impact underprivileged communities, which have been hit especially hard by the global pandemic.
What it does
The functionalities are simple. Users can touchlessly scan item barcodes and use intuitive hand gestures to operate a self-checkout kiosk (start scanning, undo item, select payment, ask for assistance).
How we built it
The front-end component is a react native web app that provides all of the user functionalities. The front-end app queries (REST API) the back-end app running locally, which deploys the machine learning models that detect/classify hand gestures and item barcodes. We used an open source machine learning model architecture and weights to implement a state-of-the-art hand gesture recognition algorithm. We used an open source library for vision-based barcode reading.
Challenges we ran into
Sending video/image data from the front-end to the back-end was especially challenging, as react native does not have a good way of collecting frames at the desired FPS for our machine learning model.
Accomplishments that we're proud of
The gesture recognition model is a good mix between lightweight and powerful. A lot of effort went into deploying the model in an efficient enough manner for real-time gesture recognition, and the results are very good.
What we learned
We learned that proper team balance and allocation of time is critical for completing demanding projects.
What's next for Touchless Self-Checkout Kiosk
Hopefully we will be able to flesh out more comprehensive features (payment, store catalog) so that the kiosk will be fully functional for general use in any store.
Log in or sign up for Devpost to join the conversation.