apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
5 releases in the last 90 days, the latest 10 days ago, with 54 contributors on GitHub.
Peaked at #80 in the weekly ranking the week of Oct 5, its best in the past year.
Use Cases & Benefits
- Provides a Python library for web scraping and browser automation to build reliable, scalable crawlers with support for headless and headful modes.
- Offers unified HTTP and headless browser crawling with automatic retries, proxy rotation, session management, and persistent queue storage for robust scraping.
- Use for extracting data from static HTML pages efficiently using BeautifulSoupCrawler without browser overhead.
- Use for scraping JavaScript-heavy websites requiring client-side rendering with PlaywrightCrawler's headless browser automation.
- Use for building asynchronous, type-safe web crawlers that integrate easily into Python applications with advanced error handling and data storage options.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
👉 View full documentation, guides and examples on the Crawlee project website 👈
We also have a TypeScript implementation of the Crawlee, which you can explore and utilize for your projects. Visit our GitHub repository for more information Crawlee for JS/TS on GitHub.
We recommend visiting the Introduction tutorial in Crawlee documentation for more information.
Crawlee is available as
crawleepackage on PyPI. This package includes the core functionality, while additional features are available as optional extras to keep dependencies and package size minimal.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
816+35
Open issues
101+21
Compared with peers in Automation
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| vercel-labs/agent-desktop | 17.0K | +13.3K | 33 | today |
| D4Vinci/Scrapling | 98.1K | +6.9K | 65 | 1 day ago |
| Panniantong/Agent-Reach | 109.3K | +3.0K | 60 | 4 days ago |
| lexmount/moli | 22.7K | +1.2K | 55 | 2 days ago |
| superplanehq/superplane | 8.8K | +435 | 60 | 7 days ago |
| apify/crawlee-python | 9.6K | — | 59 | 10 days ago |
| Category median | 22.7K | +3.0K | 60 | — |
Also by apify
Other repositories from this maintainer that GStars tracks
- TypeScript26.4K stars
crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- TypeScript13.3K stars
apify-mcp-server
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
- Python2.5K stars
agent-skills
Collection of Apify agent skills
Discover Repositories
Search across tracked repositories by name or description