• 

  JobSpy Logo

Integrating JobSpy

What is JobSpy

JobSpy is an open-source comprehensive job scraping library designed to extract job postings from multiple platforms, including LinkedIn, Indeed, Glassdoor, and ZipRecruiter. It aggregates the scraped data into a Pandas DataFrame, making it easy to manipulate and analyze. The library also supports proxy usage, enhancing its scraping capabilities and allowing for better handling of requests across different job boards.

How my Project Uses it

In my project, I utilize JobSpy to scrape job listings specifically from Indeed. The library leverages the publicly available Indeed API to gather various details such as job titles, company names, descriptions, and posting dates based on specified search queries. This information is then processed using a language model (LLM) to calculate candidate compatibility scores based on the job descriptions.

Example Integration

Here’s a brief overview of how I integrate JobSpy into my FastAPI application:

@app.post("/stream-llm-jobspy")
async def stream_llm_jobspy(query: str = "", location: str = "", listings: int = 1):
    jobs = scrape_jobs(
        site_name=["indeed"],
        search_term=query,
        location=location,
        results_wanted=listings,
        country_indeed=location,
        verbose=0,
    )

    crawler = LLMCrawler("", "", listings, model, resume)

    return StreamingResponse(
        crawler.infer(jobs), media_type="text/event-stream"
    )

This code snippet defines an endpoint that uses JobSpy to scrape job postings based on user input and streams the results for further processing.

async def infer(self, jobs):
    for idx in range(len(jobs)):
        description = jobs["description"].iloc[idx]
        response = self.condenser_evaluator_graph.execute_hybrid_graph(prompt=description)
        response["job_title"] = jobs["title"].iloc[idx]
        response["company"] = jobs["company"].iloc[idx]
        response["id"] = jobs["id"].iloc[idx]
        response["link"] = jobs["job_url"].iloc[idx]
        response["date"] = jobs["date_posted"].iloc[idx].strftime("%d-%m-%Y")

        yield json.dumps(response, indent=2)
        await asyncio.sleep(0.25)

This code snippet shows the internal LLM call made based on scraped job listings and the output from each job listing.

Installing

To install JobSpy, ensure you have Python version 3.10 or higher. You can install the library using pip:

pip install -U python-jobspy

Output

When executing a scrape with JobSpy, the output is structured in a DataFrame format containing key details about each job posting. An example output might include:

This output includes essential job details such as title, company, location, salary range, and a direct link to the job posting.

Built With

• fastapi
• pandas
• python