lyogavin/airllm
AirLLM 2.8T LLM inference with single 4GB GPU
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
3 releases in the last 90 days, the latest 62 days ago, with 17 contributors on GitHub.
Peaked at #7 in the weekly ranking the week of Aug 9, its best in the past year.
Use Cases & Benefits
- Optimizes memory usage to run large language models up to 70B parameters on a single 4GB GPU without quantization or pruning.
- Enables inference of massive LLMs on low-end GPUs by decomposing models layer-wise and supporting block-wise quantization for speedup.
- Use for deploying large-scale LLM inference on commodity hardware with limited GPU memory for research or prototyping.
- Use for accelerating inference speed up to 3x via model compression while maintaining near-original accuracy in production environments.
- Use for running diverse open-source models like Llama3, Qwen, ChatGLM, and Baichuan on MacOS or Linux with minimal setup.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning. And you can run 405B Llama3.1 on 8GB vram now.
[2024/08/20] v2.11.0: Support Qwen2.5
[2024/08/18] v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work!
[2024/07/30] Support Llama3.1 405B (example notebook). Support 8bit/4bit quantization.
[2024/04/20] AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU.
[2023/12/25] v2.8.2: Support MacOS running 70B large language models.
[2023/12/20] v2.7: Support AirLLMMixtral.
[2023/12/20] v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model.
[2023/12/18] v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement.
[2023/12/03] added support of ChatGLM, QWen, Baichuan, Mistral, InternLM!
[2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard.
[2023/12/01] airllm 2.0. Support compressions: 3x run time speed up!
[2023/11/20] airllm Initial version!
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+11,534 stars over 90 days across 23 snapshot days; best day +799 on Oct 13.
Forks
4.3K+1.1K
Open issues
152+16
Compared with peers in AI
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 8, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| D4Vinci/Scrapling | 101.1K | +6.3K | 75 | 6 days ago |
| thedotmack/claude-mem | 115.8K | +4.7K | 75 | today |
| heygen-com/hyperframes | 82.3K | +4.0K | 75 | today |
| harry0703/MoneyPrinterTurbo | 142.0K | +3.8K | 75 | 3 days ago |
| github/spec-kit | 149.2K | +2.6K | 75 | today |
| lyogavin/airllm | 41.9K | — | 38 | 62 days ago |
| Category median | 115.8K | +4.0K | 75 | — |
Discover Repositories
Search across tracked repositories by name or description