AI & DevOps · Inference
LLM inference serving, in reading order
The articles on this site about serving open-weight models yourself, arranged as one path: from a local server on a laptop to disaggregated serving on Kubernetes, and on to what each token costs.
By Matthews WongReviewed on
Key takeaway
This path follows a self-hosted model from engine to bill. It starts with the engines, vLLM and SGLang, moves to disaggregated prefill and decode and shared KV caches, then to the Kubernetes layer of llm-d, KServe, Dynamo and the Gateway API Inference Extension with the routers in front of it, and ends with autoscaling on queue depth and measuring cost per token.
Reading path · 25 articles
Read in this order
Each stage builds on the one before. If you already know a stage, skip ahead; the line under each title says what that article adds.
Stages 1 / 4
Pick and run an engine
Start small, size the hardware, then learn what the two main engines do differently.
- 01AI9 min readRun Local LLMs with Ollama: Models, API, and ModelfileWhy this step: The simplest local baseline: pull an open-weight model and call it through an OpenAI-compatible API.
- 02AI/ML12 min readvllm-metal on Apple Silicon: a Local vLLM Server for MacWhy this step: A continuously batched vLLM server on a Mac, with its memory budget and limits.
- 03Cloud11 min readJakarta GPU Capacity: VRAM Sizing for Indonesian AI TeamsWhy this step: Before renting GPUs: size VRAM and KV cache first, and weigh what quantisation gives up.
- 04AI/ML12 min readvLLM 0.31 Release Notes: Preload, Model Runner V2 and the Flags That BreakWhy this step: What vLLM 0.31 changes: the weight-cache preload daemon, Model Runner V2 as default and the flags that break on upgrade.
- 05AI/ML12 min readSGLang Unified Radix Cache: Prefix Caching for Hybrid ModelsWhy this step: SGLang's approach to prefix caching, and how its cache tiers handle hybrid attention and Mamba models.
- 06AI/ML12 min readDeepSeek V4.1 Flash on vLLM vs SGLang: What the Numbers MeanWhy this step: Both engines on one model: what each benchmark measured, why the numbers differ and what to copy.
Stages 2 / 4
Disaggregated serving and the KV cache
Splitting prefill from decode, and moving or sharing the KV cache between them.
- 07AI/ML12 min readvLLM Disaggregated Prefill Decode: A NixlConnector GuideWhy this step: Split vLLM into prefill and decode with NixlConnector, and the transfer checks that catch a broken setup before you benchmark it.
- 08AI/ML12 min readvLLM NixlPushConnector: Push vs Pull KV Transfer and TTFTWhy this step: The push variant writes KV cache to decode early: how it improves TTFT, how it fails and when pull still wins.
- 09AI/ML12 min readvLLM Bi-Directional KV Transfer for Multi-Turn AgentsWhy this step: For multi-turn agents, prefill nodes pull cached blocks back from decode; the config, TTL and limits.
- 10AI/ML11 min readLMCache Multi-Process Mode: Shared KV Cache for MoE ServingWhy this step: One shared host-memory KV pool across ranks for MoE serving, and what its benchmark does and does not show.
Stages 3 / 4
Serving on Kubernetes, and the gateway in front
The Kubernetes-native stacks that schedule and route model traffic, and the gateways that sit between your users and the models.
- 11DevOps12 min readllm-d on Kubernetes: Distributed Inference and v0.10 HardeningWhy this step: How llm-d layers on vLLM and the Gateway API Inference Extension, and what its hardening release means for rollouts.
- 12DevOps11 min readGateway API Inference Extension: EPP Moves to llm-d, What NowWhy this step: The Endpoint Picker moved to llm-d: what stays upstream and how to re-point your Helm charts.
- 13AI/ML12 min readKServe LLMInferenceService Guide: llm-d, WVA and vLLMWhy this step: KServe's LLMInferenceService adds llm-d routing and autoscaling; when it beats raw vLLM Deployments.
- 14AI/ML12 min readNVIDIA Dynamo on Kubernetes: Rust EPP and v1beta1 UpgradeWhy this step: NVIDIA's stack on Kubernetes: the Rust Endpoint Picker, the CRD storage change and a blue/green migration.
- 15AI/ML12 min readOCI Model Artifacts: Faster Kubernetes Model Cold StartWhy this step: Cold start as hidden latency: OCI image volumes, modelcar and S3 downloads measured against each other.
- 16AI/ML12 min readvLLM Semantic Router v0.4: Mixture-of-Models Routing GuideWhy this step: Route one model name across a pool of models with signals and decisions, and when that beats a static choice.
- 17Backend12 min readEnvoy AI Gateway Is Now Agent Router: What 1.2 ChangesWhy this step: Envoy AI Gateway is now Agent Router: what 1.2 breaks for MCP and credentials, and the upgrade order.
- 18Security11 min readLiteLLM CVE-2026-42208: SQL Injection Fix and Gateway HardeningWhy this step: The gateway is an attack surface too: patch the SQL injection, rotate keys and harden the proxy.
Stages 4 / 4
Autoscaling and cost per token
Scale on the signal that matters, then measure what serving costs and where caching saves it.
- 19DevOps10 min readKEDA: Event-Driven Autoscaling & Scale-to-Zero on K8sWhy this step: KEDA basics: scale on queue depth, lag, Prometheus or cron instead of CPU, with the scale-to-zero trade-offs.
- 20DevOps12 min readAutoscale vLLM on Kubernetes with KEDA: Queue Depth, Not GPU %Why this step: Applies it to vLLM: queue depth and p95 triggers rather than GPU utilisation, with activation thresholds and slow scale-down.
- 21DevOps12 min readKubernetes HPA Scale to Zero for GPU Queue Workers (1.37)Why this step: Native HPA scale to zero for GPU queue workers in 1.37, the ScaledToZero trap and how it compares with KEDA.
- 22Cloud11 min readOpenCost Inference Cost per Million Tokens for Self-Hosted vLLMWhy this step: Measure what self-hosted vLLM costs per million tokens with OpenCost, and find the break-even against an API.
- 23Cloud12 min readState of FinOps 2026: What 98% Managing AI Spend MeansWhy this step: The organisation's view: what the State of FinOps survey shows about AI cost visibility, ROI and token metrics.
- 24AI9 min readLLM Prompt Caching: The Cheapest Cost Optimization You Are Not UsingWhy this step: For hosted models: how prompt caching works across Anthropic, OpenAI and Gemini, and how to write cache-friendly prompts.
- 25AI9 min readSemantic Caching for LLMs: Cut Cost on Repeat QuestionsWhy this step: Cache whole answers to repeat questions by embedding similarity, with threshold tuning and invalidation.
The AI & DevOps field guide
More from the AI & DevOps guide
The other reading paths in this section.
- MCP & Agent SecurityPrompt injection, the 2026 MCP incidents, access controls, sandboxes and vetting agent skills.
- Kubernetes for AIThe 1.36 and 1.37 upgrades, DRA and gang scheduling, Gateway API routing and agents in the cluster.
- AI SRE & Agentic OpsIncident-investigation agents, MCP servers for ops tools, agents in CI, and the IaC beneath.
- Supply Chain & Regulationnpm and PyPI worms, model signing and AIBOM, secrets in agent commits, the CRA and EU AI Act.