# LLM inference serving, in reading order

> Self-hosted LLM inference in reading order: vLLM and SGLang, disaggregated serving and KV cache, llm-d and KServe, autoscaling and cost per token.

URL: https://www.matthewswong.com/en/ai-devops/llm-inference-serving/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Reviewed on: 17 October 2026

> **Key takeaway:** This path follows a self-hosted model from engine to bill. It starts with the engines, vLLM and SGLang, moves to disaggregated prefill and decode and shared KV caches, then to the Kubernetes layer of llm-d, KServe, Dynamo and the Gateway API Inference Extension with the routers in front of it, and ends with autoscaling on queue depth and measuring cost per token.

The articles on this site about serving open-weight models yourself, arranged as one path: from a local server on a laptop to disaggregated serving on Kubernetes, and on to what each token costs.

## Read in this order

Each stage builds on the one before. If you already know a stage, skip ahead; the line under each title says what that article adds.

### Stages 1: Pick and run an engine

Start small, size the hardware, then learn what the two main engines do differently.

1. [Run Local LLMs with Ollama: Models, API, and Modelfile](https://www.matthewswong.com/en/blog/ollama-run-local-llms/): The simplest local baseline: pull an open-weight model and call it through an OpenAI-compatible API.
2. [vllm-metal on Apple Silicon: a Local vLLM Server for Mac](https://www.matthewswong.com/en/blog/vllm-metal-apple-silicon-local-llm-server/): A continuously batched vLLM server on a Mac, with its memory budget and limits.
3. [Jakarta GPU Capacity: VRAM Sizing for Indonesian AI Teams](https://www.matthewswong.com/en/blog/jakarta-gpu-capacity-indonesian-ai-startups/): Before renting GPUs: size VRAM and KV cache first, and weigh what quantisation gives up.
4. [vLLM 0.31 Release Notes: Preload, Model Runner V2 and the Flags That Break](https://www.matthewswong.com/en/blog/vllm-0-31-fast-restart-preload-weight-cache/): What vLLM 0.31 changes: the weight-cache preload daemon, Model Runner V2 as default and the flags that break on upgrade.
5. [SGLang Unified Radix Cache: Prefix Caching for Hybrid Models](https://www.matthewswong.com/en/blog/sglang-unified-radix-cache-hybrid-model-prefix-caching/): SGLang's approach to prefix caching, and how its cache tiers handle hybrid attention and Mamba models.
6. [DeepSeek V4.1 Flash on vLLM vs SGLang: What the Numbers Mean](https://www.matthewswong.com/en/blog/deepseek-v4-1-flash-serving-vllm-vs-sglang/): Both engines on one model: what each benchmark measured, why the numbers differ and what to copy.

### Stages 2: Disaggregated serving and the KV cache

Splitting prefill from decode, and moving or sharing the KV cache between them.

7. [vLLM Disaggregated Prefill Decode: A NixlConnector Guide](https://www.matthewswong.com/en/blog/vllm-disaggregated-prefill-decode-serving-guide/): Split vLLM into prefill and decode with NixlConnector, and the transfer checks that catch a broken setup before you benchmark it.
8. [vLLM NixlPushConnector: Push vs Pull KV Transfer and TTFT](https://www.matthewswong.com/en/blog/vllm-nixl-push-connector-faster-ttft-disaggregated/): The push variant writes KV cache to decode early: how it improves TTFT, how it fails and when pull still wins.
9. [vLLM Bi-Directional KV Transfer for Multi-Turn Agents](https://www.matthewswong.com/en/blog/vllm-bidirectional-kv-transfer-multi-turn-agents/): For multi-turn agents, prefill nodes pull cached blocks back from decode; the config, TTL and limits.
10. [LMCache Multi-Process Mode: Shared KV Cache for MoE Serving](https://www.matthewswong.com/en/blog/lmcache-multi-process-mode-moe-shared-kv-cache/): One shared host-memory KV pool across ranks for MoE serving, and what its benchmark does and does not show.

### Stages 3: Serving on Kubernetes, and the gateway in front

The Kubernetes-native stacks that schedule and route model traffic, and the gateways that sit between your users and the models.

11. [llm-d on Kubernetes: Distributed Inference and v0.10 Hardening](https://www.matthewswong.com/en/blog/llm-d-kubernetes-native-distributed-inference/): How llm-d layers on vLLM and the Gateway API Inference Extension, and what its hardening release means for rollouts.
12. [Gateway API Inference Extension: EPP Moves to llm-d, What Now](https://www.matthewswong.com/en/blog/gateway-api-inference-extension-epp-moves-to-llm-d/): The Endpoint Picker moved to llm-d: what stays upstream and how to re-point your Helm charts.
13. [KServe LLMInferenceService Guide: llm-d, WVA and vLLM](https://www.matthewswong.com/en/blog/kserve-llminferenceservice-llm-d-production-guide/): KServe's LLMInferenceService adds llm-d routing and autoscaling; when it beats raw vLLM Deployments.
14. [NVIDIA Dynamo on Kubernetes: Rust EPP and v1beta1 Upgrade](https://www.matthewswong.com/en/blog/nvidia-dynamo-1-5-rust-endpoint-picker-kubernetes/): NVIDIA's stack on Kubernetes: the Rust Endpoint Picker, the CRD storage change and a blue/green migration.
15. [OCI Model Artifacts: Faster Kubernetes Model Cold Start](https://www.matthewswong.com/en/blog/oci-model-artifacts-kserve-faster-model-cold-start/): Cold start as hidden latency: OCI image volumes, modelcar and S3 downloads measured against each other.
16. [vLLM Semantic Router v0.4: Mixture-of-Models Routing Guide](https://www.matthewswong.com/en/blog/vllm-semantic-router-v0-4-mixture-of-models-routing/): Route one model name across a pool of models with signals and decisions, and when that beats a static choice.
17. [Envoy AI Gateway Is Now Agent Router: What 1.2 Changes](https://www.matthewswong.com/en/blog/envoy-ai-gateway-becomes-agent-router-1-2/): Envoy AI Gateway is now Agent Router: what 1.2 breaks for MCP and credentials, and the upgrade order.
18. [LiteLLM CVE-2026-42208: SQL Injection Fix and Gateway Hardening](https://www.matthewswong.com/en/blog/litellm-cve-2026-42208-sql-injection-hardening/): The gateway is an attack surface too: patch the SQL injection, rotate keys and harden the proxy.

### Stages 4: Autoscaling and cost per token

Scale on the signal that matters, then measure what serving costs and where caching saves it.

19. [KEDA: Event-Driven Autoscaling & Scale-to-Zero on K8s](https://www.matthewswong.com/en/blog/keda-event-driven-autoscaling-kubernetes/): KEDA basics: scale on queue depth, lag, Prometheus or cron instead of CPU, with the scale-to-zero trade-offs.
20. [Autoscale vLLM on Kubernetes with KEDA: Queue Depth, Not GPU %](https://www.matthewswong.com/en/blog/autoscale-vllm-keda-queue-depth-gpu-eks/): Applies it to vLLM: queue depth and p95 triggers rather than GPU utilisation, with activation thresholds and slow scale-down.
21. [Kubernetes HPA Scale to Zero for GPU Queue Workers (1.37)](https://www.matthewswong.com/en/blog/kubernetes-hpa-scale-to-zero-gpu-queue-workers/): Native HPA scale to zero for GPU queue workers in 1.37, the ScaledToZero trap and how it compares with KEDA.
22. [OpenCost Inference Cost per Million Tokens for Self-Hosted vLLM](https://www.matthewswong.com/en/blog/opencost-inference-cost-per-million-tokens-vllm/): Measure what self-hosted vLLM costs per million tokens with OpenCost, and find the break-even against an API.
23. [State of FinOps 2026: What 98% Managing AI Spend Means](https://www.matthewswong.com/en/blog/state-of-finops-2026-ai-cost-management/): The organisation's view: what the State of FinOps survey shows about AI cost visibility, ROI and token metrics.
24. [LLM Prompt Caching: The Cheapest Cost Optimization You Are Not Using](https://www.matthewswong.com/en/blog/llm-prompt-caching-cost-optimization/): For hosted models: how prompt caching works across Anthropic, OpenAI and Gemini, and how to write cache-friendly prompts.
25. [Semantic Caching for LLMs: Cut Cost on Repeat Questions](https://www.matthewswong.com/en/blog/semantic-caching-llm-cost-reduction/): Cache whole answers to repeat questions by embedding similarity, with threshold tuning and invalidation.

## More from the AI & DevOps guide

- [MCP & Agent Security](https://www.matthewswong.com/en/ai-devops/mcp-agent-security/): Prompt injection, the 2026 MCP incidents, access controls, sandboxes and vetting agent skills. (25 articles)
- [Kubernetes for AI](https://www.matthewswong.com/en/ai-devops/kubernetes-for-ai/): The 1.36 and 1.37 upgrades, DRA and gang scheduling, Gateway API routing and agents in the cluster. (24 articles)
- [AI SRE & Agentic Ops](https://www.matthewswong.com/en/ai-devops/ai-sre-and-agentic-ops/): Incident-investigation agents, MCP servers for ops tools, agents in CI, and the IaC beneath. (25 articles)
- [Supply Chain & Regulation](https://www.matthewswong.com/en/ai-devops/software-supply-chain-and-regulation/): npm and PyPI worms, model signing and AIBOM, secrets in agent commits, the CRA and EU AI Act. (23 articles)

---

- HTML version: https://www.matthewswong.com/en/ai-devops/llm-inference-serving/
- Bahasa Indonesia: https://www.matthewswong.com/id/ai-devops/llm-inference-serving/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json