open-compass/VLMEvalKit
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
No releases in the last 90 days; the most recent was 500 days ago, with 299 contributors on GitHub.
Gained 31% fewer stars in the last 90 days than in the 90 days before.
Use Cases & Benefits
- Provides an open-source Python toolkit for evaluating large vision-language models across 80+ benchmarks and 220+ models with one-command execution.
- Simplifies multi-modal model evaluation by automating data preparation, inference, and metric calculation, enabling reproducible and generation-based assessments.
- Use for benchmarking new vision-language models on diverse image and video datasets to compare performance with state-of-the-art models.
- Use for integrating multi-node distributed inference to accelerate evaluation workflows of large or complex multi-modal models.
- Use for developing and contributing custom benchmarks or model evaluation pipelines within a unified, extensible framework.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
A Toolkit for Evaluating Large Vision-Language Models.
VLMEvalKit (the python package name is vlmeval) is an open-source evaluation toolkit of large vision-language models (LVLMs). It enables one-command evaluation of LVLMs on various benchmarks, without the heavy workload of data preparation under multiple repositories. In VLMEvalKit, we adopt generation-based evaluation for all LVLMs, and provide the evaluation results obtained with both exact matching and LLM-based answer extraction.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
787+42
Open issues
325+40
Compared with peers in AI
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| thedotmack/claude-mem | 113.6K | +8.8K | 35 | today |
| D4Vinci/Scrapling | 98.1K | +6.9K | 65 | 1 day ago |
| Leonxlnx/taste-skill | 110.7K | +5.2K | 55 | 1 day ago |
| calesthio/OpenMontage | 77.2K | +4.2K | 40 | — |
| heygen-com/hyperframes | 79.6K | +4.1K | 45 | today |
| open-compass/VLMEvalKit | 4.5K | — | 35 | 500 days ago |
| Category median | 98.1K | +5.2K | 45 | — |
Discover Repositories
Search across tracked repositories by name or description