apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
4 releases in the last 90 days, the latest 30 days ago, with 149 contributors on GitHub.
Use Cases & Benefits
- Provides a Node.js library for web scraping and browser automation using JavaScript and TypeScript.
- Enables reliable, human-like crawling with integrated proxy rotation and support for multiple browser engines and HTTP clients.
- Use for extracting structured data and files from websites with headless or headful browser automation.
- Use for building scalable crawlers that handle complex JavaScript-rendered pages using Playwright or Puppeteer.
- Use for automating data collection workflows that require persistent URL queues and customizable crawling lifecycles.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.
Crawlee is available as the
crawleeNPM package.👉 View full documentation, guides and examples on the Crawlee project website 👈
Crawlee for Python is open for early adopters. 🐍 👉 Checkout the source code 👈.
We recommend visiting the Introduction tutorial in Crawlee documentation for more information.
Crawlee requires Node.js 16 or higher.
The fastest way to try Crawlee out is to use the Crawlee CLI and choose the Getting started example. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+1,022 stars over 88 days across 9 snapshot days; best day +10 on Oct 10.
Forks
1.7K+136
Open issues
130-18
Compared with peers in Automation
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 8, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| vercel-labs/agent-desktop | 21.4K | +8.7K | 75 | 5 days ago |
| D4Vinci/Scrapling | 101.1K | +6.3K | 75 | 6 days ago |
| harry0703/MoneyPrinterTurbo | 142.0K | +3.8K | 75 | 3 days ago |
| superplanehq/superplane | 9.8K | +1.0K | 75 | 13 days ago |
| gosom/google-maps-scraper | 7.9K | +622 | 75 | 2 days ago |
| apify/crawlee | 26.4K | — | 58 | 30 days ago |
| Category median | 21.4K | +3.8K | 75 | — |
Also by apify
Other repositories from this maintainer that GStars tracks
- TypeScript15.1K stars
apify-mcp-server
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
- Python9.7K stars
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
- Python2.5K stars
agent-skills
Collection of Apify agent skills
Discover Repositories
Search across tracked repositories by name or description