Webscraping with python is a field I feel comfortable to work with. As a data science aspirant, Data mining is a must know technique which is the primary reason which inspired me to take on the challenge of webscraping a site. I come from a non-C.S background so learning python as my first language was a venture but I got along and now, looking at all the fields python leads one to is just mesmerizing. I have tried to make it as presentable as possible, my very first project which can scrape and store the data for analysis.
What it does
Scrappy --v 1.2 scrapes the item you wish to search from amazon, provides you the first 21 items of a page in a csv file without you having to open the site and search manually. For a competitive environment it can be used to scrape data from a competitors site, analyze and compare it to boost your product.
## How I built it
I have used Python along with Jupyter notebook to analyze, the program uses selenium to automate Chrome and Beautiful Soup to scrape the data from the page. I used Pandas to analyze and visualize the data in jupyter notebook. I built Scrappy --v 1.2 inspired by the mother project Scrappy which I made myself and is available in my github repo although, they are completely different from each other.
## Challenges we ran into
I am very new to Pandas, I couldn't do much with the library. I ran into a halt as I could not get the link to my data file, although it was just a small error where (URL + link) was needed.
I ran into another problem where i tried to implement matplotlib but couldn't.
## Accomplishments that I am proud of
I finished the project, I made it somehow or somewhat representable.
Taking part in this wonderful week of Build with MLH inspired me a lot in many different ways.
I am proud that i took part in the Hackathon.
## What I learned
I learnt more about webscraping with Python and its libraries such as bs4, selenium, requests.
I learnt to work in a timely fashion.
I learnt about so many technologies and sites that are available for us in the web for free.
## What's next for Scrappy --v 1.2
I would like to add in some more sites to analyze and compare themselves. I will add a visual data representation and show the hikes in price , comparison of price , reviews with different sites.
Adding other sites like sport team comparison and analysis is what I am looking forward at.
Log in or sign up for Devpost to join the conversation.