OSS Radar projects in the model serving category.
Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
Why: +507 stars in 7 days; 65 commits in 30 days
deepseek gemma gemma3 glm go golang
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
Why: +1,037 stars in 7 days; 100 commits in 30 days
ai llm llm-ui llm-webui llms mcp
✨ Light and Fast AI Assistant. Support: Web | iOS | MacOS | Android | Linux | Windows
Why: +180 stars in 7 days; 44 commits in 30 days
calclaude chatgpt claude cross-platform desktop fe
A high-throughput and memory-efficient inference and serving engine for LLMs
Why: +659 stars in 7 days; 100 commits in 30 days
amd blackwell cuda deepseek deepseek-v3 gpt
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Why: 100 active contributors
ai-chat llm-inference
RDNA-native LLM inference engine in Rust.
Why: +8 stars in 7 days; 100 commits in 30 days
amd-gpu gpu-computing hip llm-inference machine-learning quantization
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, ch
Why: +10 stars in 7 days; 100 commits in 30 days
anthropic anthropic-api chat claude gemini inference-gateway
Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
Why: +3 stars in 7 days; 95 commits in 30 days
anthropic data-labeling deepinfra domain-specific-language experiments llama2
LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, suppo
Why: +4 stars in 7 days; 100 commits in 30 days
cpu decode gpu hybrid inference model
The open creative AI workspace
Why: +8 stars in 7 days; 100 commits in 30 days
anthropic cerebras cohere deepseek fal gemini
Local AI Assistant on your phone
Why: +11 stars in 7 days; 89 commits in 30 days
ai gemma3 gemma3n gemma4 gemma4-agent-skills gptoss
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
Why: +5 stars in 7 days; 23 commits in 30 days
deepseek k8s kimi-k2 llama llm llm-inference
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
Why: +10 stars in 7 days; 100 commits in 30 days
agentic-ai kv-cache llm llm-inference long-context quantization
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
Why: +3 stars in 7 days; 40 commits in 30 days
cpu-inference gguf-model-support gpu-inference high-performance-inference hybrid-inference inference-engine
sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems
Why: +15 stars in 7 days; 13 commits in 30 days
dgx-spark inference llama-cpp sglang vllm
The Runpod worker template for serving our large language model endpoints. Powered by vLLM.
Why: +3 stars in 30 days; 27 commits in 30 days
language-model llm runpod vllm
An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.
Why: +3 stars in 7 days
artificial-intelligence dynamo huggingface kubernetes llm llm-inference
Python package for LLM compression
Why: +3 stars in 7 days; 10 active contributors
llm qep quantization vllm
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Why: 100 active contributors
ai-chat llm-inference