Web Scraping is an automatic way to retrieve unstructured data from a website and store them in a structured format. For example, if you want to analyze what kind of face mask can sell better in Singapore, you may want to scrape all the face mask information on an E-Commerce website like Lazada. Scraping makes the website traffic spike and may cause the breakdown of the website server. Thus, not all websites allow people to scrape.
How does it works?
Web scraping just works like a bot person browsing different pages website and copy pastedown all the contents. When you run the code, it will send a request to the server and the data is contained in the response you get. What you then do is parse the response data and extract out the parts you want.
How do we scrap?
- Approach 1: If website stores all their information on the HTML front end, you can directly use code to download the HTML contents and extract out useful information.
- Approach 2: If website stores data in API and the website queries the API each time when user visit the website, you can simulate the request and directly query data from the API
Built With
- blog
- python
- webscraping
Log in or sign up for Devpost to join the conversation.