Walmart Recommendation System

Built a multi-threaded web crawler to scrape data related to products of different categories and implemented a recommendation system for the products.

Comment

Inspiration:

Efficient recommendation systems make a big difference to any consumers' experience on the platform. Graph based Knowledge models was our go to attempt at making one such recommendation system

What it does:

We have developed a multi-threaded web crawler which scrapes for data from 9 categories on the Walmart website with visiting around 12,500 webpages altogether. We get data like, Product Ids, Product Name, Descriptions, Ratings, Similar products as the raw data. We then index the raw data to general categories and then use this to build a graph based knowledge model to act as a Recommendation system.

How I built it

The challenge was divided into sub tasks. We spend a significant time scraping the data and finally realized we could parallelize it. It was much faster and we could collect around 12,000 data points. Based on the recommendations available on the site, we built a recommender system to build a recommendation system "just" based on graph connectivity metrics and not using any of the semantics or keywords. We also built a categorization algorithm to find out what group a particular search query belonged to.

Challenges I ran into

We have initially ran into challenges pertaining to data collection. the first task itself took a lot of time and the next challenge was to build the graph based recommendation model from scratch.

Accomplishments that I'm proud of

Designing a Multi-threaded web-crawler which could get around 15k data-points in just 1 hr of scraping for the first time gave a great sense of achievement,

What I learned

While developing the web crawler, we learnt how to read the XML structure of the page and develop your scraping script accordingly. There are a lot of custom tags included in the Walmart Page which we needed to extract for information. We also learnt that crawling can be an expensive process (in terms of computational resources and it is key to optimize those processes). We used multithreaded web crawling functions for the same.

For creating the classification algorithm, we learnt about how to train and tune a model for accurate category classification for a product. To be able to develop a functioning algorithm within a small span of time, it is better to use built in dataset tools instead of tweaking and dealing with the data types/sizes.

We also learnt about graph algorithms and used Gephi to analyze, study and visualize the graph data for the recommender systems.

Lastly we learnt on how to deploy all our ML algorithms as an application using a very simple front-end. This completed the pipeline of a data-science project: from data gathering to deploying an app for us.

What's next for Walmart Recommendation System

We should build a better front-end interface for the consumption of the model. Also, the models can always be revised for better performance

Video for individual demos:

Built With

Share this project:

Updates