llm-d/llm-d-router
llm-d Router: The intelligent entry point for inference requests
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
Ranks #109 of 2,531 repositories GStars tracks this week (top 4%).
5 releases in the last 90 days, the latest 5 days ago, with 293 contributors on GitHub.
Peaked at #55 in the weekly ranking the week of Oct 11, its best in the past year.
Use Cases & Benefits
- Delivers intelligent routing and flow control for LLM inference requests using an Endpoint Picker integrated with proxies like Envoy.
- Optimizes request placement by considering load, cache locality, and priority, enabling efficient and flexible inference traffic management.
- Use for deploying scalable LLM inference gateways in Kubernetes environments with advanced traffic and load balancing.
- Use for managing multi-stage inference workflows with disaggregation sidecars coordinating complex prefill and decode stages.
- Use for integrating with L7 load balancers and cloud-managed services to unify inference and traditional workload routing.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
- Code of conduct: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
The llm-d Router is the intelligent entry point for inference traffic, delivering LLM load and prefix-cache aware routing, request prioritization, and advanced flow control across diverse request formats to fulfill complex serving objectives. It supports a flexible deployment model: it can run in Standalone Mode, where a self-managed Envoy proxy runs either alongside the EPP or as a separate service, or integrate with L7 load balancers—including self-managed instances (e.g., Istio, AgentGateway) and cloud-managed services (e.g., Google Cloud's Application Load Balancer)—via the Kubernetes Gateway API.
The router achieves its intelligence through an Endpoint Picker (EPP) that integrates with production-grade proxies (such as Envoy) via the ext-proc protocol, injecting real-time signals into the data plane to optimize request placement.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+54 stars over 16 days across 16 snapshot days; best day +8 on Oct 10.
Forks
476+58
Open issues
381+41
Compared with peers in AI
Fastest-growing repositories GStars tracks in this cohort for the week of Oct 18, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| storytold/artcraft | 35.2K | +8.1K | 75 | 25 days ago |
| heygen-com/hyperframes | 69.2K | +5.7K | 45 | today |
| Comfy-Org/ComfyUI | 141.5K | +3.9K | 60 | 2 days ago |
| ifixai-ai/iFixAi | 30.7K | +3.3K | 75 | 5 days ago |
| harry0703/MoneyPrinterTurbo | 133.8K | +3.1K | 75 | today |
| llm-d/llm-d-router | 428 | +26 | 65 | 5 days ago |
| Category median | 69.2K | +3.9K | 75 | — |
Discover Repositories
Search across tracked repositories by name or description