GStars
    crawlab-team

    crawlab-team/crawlab

    Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

    devops
    crawlab
    crawler
    crawling-tasks
    docker
    go
    Go
    BSD-3-Clause
    12.3K stars
    1.9K forks
    210 GitHub watchers
    Updated Nov 9, 2026
    View on GitHub

    Build with Backblaze B2

    SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.

    Explore Backblaze Labs

    Loading star history...

    Where it stands

    Rank, growth pace and release cadence relative to everything GStars tracks.

    No releases in the last 90 days; the most recent was 878 days ago, with 28 contributors on GitHub.

    Use Cases & Benefits

    • Provides a distributed web crawler management platform supporting spiders in any language and framework.
    • Enables language-agnostic spider management with a scalable architecture and a user-friendly web UI.
    • Use for orchestrating and scaling large-scale web crawling tasks across multiple worker nodes.
    • Use for integrating and managing spiders built with diverse frameworks like Scrapy, Puppeteer, or Selenium.
    • Use for monitoring, scheduling, and analyzing crawling tasks with real-time logs and result visualization.

    Should you depend on it?

    License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.

    License

    BSD 3-Clause

    Permissive: commercial use, modification and redistribution are allowed with attribution.

    Community files

    • README: present

    Watch-outs

    • No commits in the last six months (last push Feb 2026).

    General guidance from repository metadata, not legal advice. How we score

    From the README

    Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium.

    You can follow the installation guide.

    Please open the command line prompt and execute the command below. Make sure you have installed docker-compose in advance.

    Next, you can look into the docker-compose.yml (with detailed config params) and the Documentation for further information.

    Excerpt from the crawlab-team/crawlab README on GitHub (BSD-3-Clause license). © its authors.Read the full README on GitHub

    Daily snapshots

    Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots

    Forks

    1.9K-2

    Open issues

    166+0

    Compared with peers in DevOps

    Fastest-growing repositories GStars tracks in this cohort for the week of Nov 8, by 7-day star growth

    View category →
    crawlab-team/crawlab compared with DevOps repositories by stars, 7-day star growth, health score and last release
    RepositoryStars+7dHealthLast release
    milind-soni/OpenMausBot20.6K+8.4K75today
    DuarteSantos8/openGym20.8K+1.7K757 days ago
    superplanehq/superplane9.8K+1.0K7513 days ago
    elie222/rakazo7.8K+95875today
    AstrBotDevs/AstrBot46.6K+7747510 days ago
    crawlab-team/crawlab12.3K—22878 days ago
    Category median20.6K+1.0K75—
    Compare crawlab with OpenMausBot, openGym, superplane →

    Discover Repositories

    Search across tracked repositories by name or description