Inspiration
Web scraping was a term that thrown left and right, from my perspective. I would see it on occasion, but I would be a bit intimidated by it. I thought, "What does scraping mean? How do you scrape the web? Am I going to be dealing with large datasets? I've seen weird websites that contain thousands of Reddit posts, do those websites use web scrapers? Is it illegal? Am I going to have the FBI knock?"
But today, one of the challenges was to create a web scraper. So I just sat down, looked up a tutorial, and got a pleasant surprise.
What it does
Jobscrape is technically a web scraper, but it's more of a beginner project instead of an actual, applicable, flexible, usable web scraper that deals with real web information. Jobscrape scrapes this fake job website, which is part of the tutorial found here.
How I built it
Jobscrape was built using Python (requests module and bs4 module), along with the tutorial from Real Python linked in the above section.
Challenges I ran into
There were some ups and downs, that's for sure. I tried to tackle the problem of sorting through the scraped code myself before looking at the tutorial, but since this was my first time actually using Beautiful Soup (bs4), needless to say, I made a mess. But I picked myself up, read through the tutorial, and made a mini program.
Accomplishments that I'm proud of
I'm proud of accomplishing the project that I've been putting off for a long time (I think I waited a year, maybe two?).
What I learned
I learnt the bare basics of web scraping, as well as how the Beautiful Soup module (is it library or module? I keep getting it mixed up in these posts), and I also learnt that web scraping isn't really all that intimidating, and it's useful at times.
Log in or sign up for Devpost to join the conversation.