smg-project/smg
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
Build with Backblaze B2
SDKs, agent skills, IDE extensions, and reference pipelines from Backblaze Labs. All open source.
Loading star history...
Where it stands
Rank, growth pace and release cadence relative to everything GStars tracks.
5 releases in the last 90 days, the latest 17 days ago, with 160 contributors on GitHub.
Use Cases & Benefits
- Provides an engine-agnostic, high-performance gateway for routing large language model requests across diverse backends via a unified API.
- Enables efficient GPU utilization and enterprise-grade multi-tenant control with cache-aware routing, gRPC streaming, and extensible WebAssembly plugins.
- Use for centralizing and load-balancing LLM inference across self-hosted and cloud providers in scalable production environments.
- Use for implementing secure multi-tenant LLM services with fine-grained API key authentication and chat history management within private infrastructure.
- Use for integrating heterogeneous LLM engines and APIs like OpenAI, Anthropic, Gemini, and custom endpoints under a single, observable gateway.
Should you depend on it?
License terms, project hygiene and risks, from GitHub metadata and GStars' snapshots.
License
Apache 2.0
Permissive: commercial use, modification and redistribution are allowed with attribution; includes an explicit patent grant from contributors.
Community files
- README: present
- Contributing guide: present
- Code of conduct: present
Watch-outs
No watch-outs from GStars' checks.
General guidance from repository metadata, not legal advice. How we score
From the README
Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.
API Coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs.
Install — pick your preferred method:
Run — point SMG at your inference workers:
That's it. SMG is now load-balancing requests across your workers.
Daily snapshots
Stars gained per day, and how forks and open issues moved, from GStars' daily snapshots
Forks
204+9
Open issues
188+27
Compared with peers in LLM
Fastest-growing repositories GStars tracks in this cohort for the week of Nov 1, by 7-day star growth
| Repository | Stars | +7d | Health | Last release |
|---|---|---|---|---|
| thedotmack/claude-mem | 113.6K | +8.8K | 35 | today |
| NVIDIA/auto-ontology | 7.5K | +5.6K | 37 | — |
| HKUDS/DeepTutor | 55.5K | +5.5K | 65 | 5 days ago |
| affaan-m/ECC | 297.9K | +5.2K | 63 | 35 days ago |
| DietrichGebert/ponytail | 185.3K | +5.1K | 60 | 7 days ago |
| smg-project/smg | 696 | — | 64 | 17 days ago |
| Category median | 113.6K | +5.5K | 60 | — |
Discover Repositories
Search across tracked repositories by name or description