Comparison shopping for the best prices has never been quick or easy. No two brick-and-mortar stores have exactly the same items or the same prices. Online shopping is even worse, with dozens of e-stores offering even more combinations of options and prices. It’s a difficult and time-consuming process to find exactly what you need at the lowest price.
Then COVID-19 came along and made everything tougher. Massive layoffs have made personal budgets tighter than ever before. Store closures and supply chain disruptions have made things even harder to find. And with the number of new COVID-19 cases skyrocketing in many parts of the country, many people simply don’t want to take the risk of going into a store even if it does have what they want.
What it does
Shopper Bot is a one-stop online shopping hub that scours the internet to find exactly what you need at the lowest prices, while letting you stay safe at home. By scraping different retails' websites, Shopper Bot helps you find the lowest price for the item you're searching for, without needing to do the comparison shopping yourself. It also helps you filter out items that are sold out, so you don't need to waste your time filtering out listings of products you can't even buy.
How we built it
The entire project was built in UiPath. Basically, we built 6 different modules: an input module, 2 store search modules, 2 store scraper modules, and an output module. The user first enters the search term that they are looking for, then that gets handed off to each of the store search modules (Walmart and Amazon at the moment). Those then analyze the search results pages to extract all of the product page links, which then get handed to the store scraper modules. Each of those modules then opens each product link, extracts the relevant information like product name, price, and if there is a product image, then all of that data is standardized, packed, and set to the output module. From there, the output module combines the data sent by the scrapers and a pre-built website template that handles everything from sorting to filtering to data display, and final that gets loaded into a document for the user to save. The only thing the user needs to do after entering what they are searching for is save that document as a web page, and open it in their browser.
Challenges we ran into
We had a hard time scraping the data from the internet using UiPath. While setting the search up was easy enough, trying to get it to parse through each of the different search results was much harder. We needed to basically write our own HTML DOM tree parser to process the results once we got the search container to use UiPath.
Accomplishments that we're proud of
This was our first time creating a web scraper, and we were really proud of the fact that we got everything completely working!
What we learned
We learned a ton about web scraping, particularly with how to select different elements of the web page that may occur in different numbers or not at all depending on the search results.
What's next for Shopper Bot
In the future, we’d like to expand the comparison shopping to other stores, such as Kroger, Costco, Target, etc. We'd also like to add more functionality to both the scraper and the front end to do things like filter by brand or pull and display the product reviews