Crawlee library

Crawlee is an open-source library for web scraping and browser automation. It allows to quickly scrape data, store it, and avoid getting blocked, headless browsers, and smart proxy rotation.

Comment

Hi! 👋 Today we’re launching Crawlee, an open source web scraping and automation library for Node.js 🚀

This project really is a culmination of 4 years of work trying to make the best library for web scraping in production. Web scraping is a very dynamic environment and what works today might not work tomorrow, so we at Apify had to go through a lot of trial and error to figure out the most reliable and convenient ways of crawling the web and scraping data. We hope that we finally cracked it and that many developers will enjoy working with it and it will make their scrapers more reliable and time to production faster. We value our community as without them, we would not be able to find bugs so quickly and improve even our own production scrapers. Finally, the full list of Crawlee's features:

🛠 Features​

  • Single interface for HTTP and headless browser crawling
  • Persistent queue for URLs to crawl (breadth & depth first)
  • Pluggable storage of both tabular data and files
  • Automatic scaling with available system resources
  • Integrated proxy rotation and session management
  • Lifecycles customizable with hooks
  • CLI to bootstrap your projects
  • Configurable routing, error handling and retries
  • Dockerfiles ready to deploy
  • Written in TypeScript with generics

👾 HTTP crawling​

  • Zero config HTTP2 support, even for proxies
  • Automatic generation of browser-like headers
  • Replication of browser TLS fingerprints
  • Integrated fast HTML parsers. Cheerio and JSDOM
  • Yes, you can scrape JSON APIs as well

💻 Real browser crawling​

  • JavaScript rendering and screenshots
  • Headless and headful support
  • Zero-config generation of human-like fingerprints
  • Automatic browser management
  • Use Playwright and Puppeteer with the same interface
  • Chrome, Firefox, Webkit and many others

If you have any questions about this library, you can find us on GitHub, Discord, anywhere really. Cheers!

Built With

Share this project:

Updates