kvcache-ai/ktransformers
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
Ranks #4 among LLM repositories tracked this week, and #4 over the last 30 days; #597 of 2,794 repositories GStars tracks this week (top 21%).
5 releases in the last 90 days, the latest 4 days ago, with 121 contributors on GitHub.
Peaked at #28 in the weekly ranking the week of Nov 30, its best in the past year.
Use Cases & Benefits
- Provides a flexible Python framework to optimize and accelerate large language model inference and fine-tuning with advanced kernel and parallelism techniques.
- Enables significant speedups and memory efficiency on local heterogeneous hardware by injecting optimized modules with minimal code changes.
- Use for experimenting with and deploying resource-efficient LLM inference on local desktops with limited GPU/CPU resources.
- Use for integrating optimized LLM backends compatible with OpenAI and Ollama APIs into custom applications or VSCode extensions.
- Use for customizing and combining multiple kernel optimizations via YAML templates to tailor performance for specific transformer models.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
- Code of conduct: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations 🌟 Show Cases | 🚀 Quick Start | 📃 Tutorial | 💬 Discussion|🙋 FAQ
KTransformers, pronounced as Quick Transformers, is designed to enhance your 🤗 Transformers experience with advanced kernel optimizations and placement/parallelism strategies.
KTransformers is a flexible, Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a Transformers-compatible interface, RESTful APIs compliant with OpenAI and Ollama, and even a simplified ChatGPT-like web UI.
Our vision for KTransformers is to serve as a flexible platform for experimenting with innovative LLM inference optimizations. Please let us know if you need any other features.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+703 stars over 90 days across 14 snapshot days; best day +46 on Jul 24.
Forks
1.6K+128
Open issues
524+57
Compared with peers in LLM
Fastest-growing repositories GStars tracks in this cohort for the week of Oct 18, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| DietrichGebert/ponytail | 173.5K | +9.2K | 75 | 6 days ago |
| xerj-org/xerj | 13.1K | +8.1K | 75 | today |
| tashfeenahmed/freellmapi | 39.6K | +3.9K | 75 | 1 day ago |
| JuliusBrussee/caveman | 114.4K | +2.2K | 75 | 9 days ago |
| thesysdev/openui | 13.1K | +2.1K | 75 | 7 days ago |
| kvcache-ai/ktransformers | 19.6K | +0 | 75 | 4 days ago |
| Category median | 39.6K | +3.9K | 75 | — |
Also by kvcache-ai
Other repositories from this maintainer that GStars tracks
Discover Repositories
Search across tracked repositories by name or description