From our conversations with T-Mobile retail experts, we learned that customers might visit a retail store 7 times before buying a phone. We wanted to build an Artificial Intelligence and Machine Learning solution that would connect T-Mobile to a novel data source - foot traffic from its own stores - to help speed up and lock in these customer sales. How can we efficiently share knowledge between mobile experts to further cut down on wasted time in customer interactions? What kind of information can we track from our customers to further improve both our engagements with customers and our sales? In our hack, we leverage deep learning and computer vision to try to tackle these tough questions.
What it does
This product is a web-based monitoring tool that enables T-Mobile analysts and sales reps to build insights from customer behavior in retail stores, enabling them to push the limits of customer experience. By recording footage to track interactions and movements in retail stores, we are able to analyze this data using deep learning and computer vision to uncover what kinds of brands and products specific customers care about. Our models detect and track pedestrian movements by performing real-time object detection.
Since we know the layouts of T-Mobile retail stores in advance, this real-time location data allows us to identify many areas of interest, such as pinpointing hottest regions of the store at a given time, or personalizing each individual customer's tendency towards each part of the store. With these highly actionable insights, we can empower mobile experts in order to understand their customers even before their first interaction. At the store level, the daily data analysis will allow us to better understand buying patterns and shopping preferences within each region, and tailor each individual retail store to their frequent customers and optimize store layouts.
At the personal level, understanding customer foot patterns will allow mobile experts to serve customers more efficiently by having an idea of customer preferences in advance, so they can be prepared with more relevant information and questions that the customer may have. Furthermore, since mobile experts can change from day to day, we can encode the past interactions and tendencies of each customer to efficiently share knowledge between mobile experts and cut down on the wasted time repeating past conversations with other representatives. Moreover, this enables us to send customers targeted sales and advertisements based on their foot patterns to increase customer interest and sales.
How we built it
By using a Raspberry Pi camera module, we are able to record real time data that represents 'customers' moving around a 'retail store'. By using the state-of-the art deep learning framework YOLOv3, we were able to perform object detection to pinpoint and locate customers in our store, and then track their movements around the store using Simple Online and Realtime Tracking with a Deep Association Metric, more commonly known as Deep SORT, both of these which leverage the machine learning frameworks TensorFlow and Keras.
After aggregating this data, we used D3.js to generate heat maps and pie charts to convey both store-level and customer-level insights in a Flask web app.
Challenges we ran into
Unfortunately, as powerful as machine learning can be, highly complex deep learning requires extremely powerful computers souped up with multiple GPUs. With only our laptops to work with during the hackathon, and limited time to create such models, we were unable to delve into complex deep learning models that would potentially take multiple days or weeks to train. Furthermore, the only mountable camera we were able to use was the Raspberry Pi camera module, and as convenient as the Pi can be, it does not generate extremely high quality videos, further adding to our hardware limitations.
Additionally, we did not have actual access to real retail store surveillance data, causing us to need to generate this data ourselves by simulating a retail environment, and to generate our analytics and insights based on this data. With actual retail store surveillance data, we would be able to generate the real insights from real customers.
Accomplishments that we're proud of
We were able to create a fully functional and clean web application that provides the user a lot of useful information about patterns in customer behavior.
What's next for MagentaGo
Ideally, we would want to leverage our models on the actual retail store data in order to extract the insights for real retail stores and customer data. Furthermore, we could continue to add more features, such as integrating with more IoT services in order to leverage more customer data and generate more useful insights for the users.