vllm-project/semantic-router
An open, programmable decision layer for models and compute.
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
Ranks #54 of 2,845 repositories GStars tracks this week (top 2%).
1 release in the last 90 days, the latest 17 days ago, with 254 contributors on GitHub.
Peaked at #39 in the weekly ranking the week of Dec 21, its best in the past year.
Use Cases & Benefits
- Semantic Router is an intelligent Mixture-of-Models router that auto-selects the best LLM or LoRA adapter based on semantic understanding of requests.
- Implemented in Go with Rust FFI, it integrates with Kubernetes, supports HuggingFace models, and features domain-aware prompts and semantic caching.
- Strengths include improved inference accuracy, real-time PII detection, jailbreak prompt guard, and reduced latency via semantic caching; requires model pool setup.
- Organizations can deploy it in production for scalable, secure LLM inference routing, optimizing resource use and enhancing privacy compliance.
- Ideal for enterprises needing dynamic LLM selection for diverse tasks, secure prompt handling, and efficient multi-model AI gateway management.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
- Code of conduct: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
A Mixture-of-Models (MoM) router that intelligently directs OpenAI API requests to the most suitable models or LoRA adapters from a defined pool based on Semantic Understanding of the request's intent (Complexity, Task, Tools).
Conceptually similar to Mixture-of-Experts (MoE) which lives within a model, this system selects the best entire model for the nature of the task.
As such, the overall inference accuracy is improved by using a pool of models that are better suited for different types of tasks:
Benchmarking will be conducted to determine the best implementation.
Select the tools to use based on the prompt, avoiding the use of tools that are not relevant to the prompt so as to reduce the number of prompt tokens and improve tool selection accuracy by the LLM.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
+1,127 stars over 83 days across 43 snapshot days; best day +39 on Sep 28.
Forks
1.0K+273
Open issues
654+390
Compared with peers in Kubernetes
Fastest-growing repositories GStars tracks in this cohort for the week of Oct 11, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| Project-HAMi/HAMi | 4.8K | +199 | 75 | 56 days ago |
| seaweedfs/seaweedfs | 35.7K | +152 | 75 | today |
| google/gvisor | 19.7K | +140 | 75 | 5 days ago |
| traefik/traefik | 65.2K | +133 | 75 | 5 days ago |
| ys-ll/uniterm | 881 | +106 | 75 | 5 days ago |
| vllm-project/semantic-router | 6.2K | +0 | 75 | 17 days ago |
| Category median | 19.7K | +140 | 75 | — |
Also by vllm-project
Other repositories from this maintainer that GStars tracks
- Python95.3K stars
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
- Python7.2K stars
vllm-omni
A framework for efficient model inference with omni-modality models
- Go5.1K stars
aibrix
Cost-efficient and pluggable Infrastructure components for GenAI inference
- Python3.0K stars
vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
- JavaScript1.1K stars
recipes
Common recipes to run vLLM
Discover Repositories
Search across tracked repositories by name or description