llm-d on Kubernetes: Distributed Inference and v0.10 Hardening

A plain Kubernetes Service sends a request to whichever vLLM pod it likes, and that is a poor choice for an LLM server. One pod may already hold the conversation's prompt prefix in its KV cache, another may be buried under long prefills, and the Service cannot tell them apart. llm-d exists to fix that routing decision, and its v0.10.0 release on 29 September is the first one whose headline is making the production path boring rather than adding a feature.

This is a reading, not a benchmark. I went through the v0.8.0, v0.9.0 and v0.10.0 release notes, the CNCF project page, the architecture and quickstart docs, and the actual values files and kustomize components in the repository. I did not run it on a GPU cluster, so every number below is quoted from a file I read, and where the docs are silent I say so.

What llm-d adds on top of vLLM

The project README calls llm-d a distributed inference serving stack for Kubernetes, founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, licensed Apache 2.0. It sits above model servers such as vLLM and SGLang rather than replacing them. The useful way to read it is by ownership: who decides what.

The README groups the extras into five areas: intelligent routing, KV-cache management with tiered offloading, serving large models through prefill/decode disaggregation and wide expert parallelism, operational features such as flow control and SLO-aware autoscaling, and batch processing. The CNCF page lists the same capabilities and records that llm-d was accepted to CNCF on 12 March 2026 at Sandbox maturity, which matters for how much weight to put on it later.

The request path: Gateway, EPP and InferencePool

The Inference Extension documentation describes the flow in five steps, and llm-d's architecture page confirms the same shape with its own proxy conformant to the extension:

1. The gateway selects an InferencePool or Service using the ordinary Gateway and HTTPRoute APIs.
2. If the target is an InferencePool, the gateway sends the request information to that pool's Endpoint Picker over ext-proc.
3. The EPP gathers metrics from the pool's endpoints, possibly asynchronously.
4. The EPP tells the gateway which endpoint should receive the request.
5. The gateway routes the request to that endpoint.

An InferencePool groups the model server pods for one base model through a label selector, and the labels must exactly match those on the pods. That is the whole discovery mechanism: the router does not register backends, it watches labels. The picker is therefore a second control plane in the request path, and everything about failure behaviour later in this post follows from that.

Running the optimized baseline

The v0.10 quickstart deploys the Optimized Baseline in Standalone Mode, where the router chart includes its own Envoy proxy, so no separate gateway is needed. Prerequisites are kubectl, helm, GPU nodes for the default vLLM and NVIDIA configuration, and a HuggingFace token. These are the steps as the page gives them, condensed:

# The v0.10 quickstart (Optimized Baseline, Standalone Mode), condensed.
git clone https://github.com/llm-d/llm-d.git && cd llm-d
git checkout main            # I would check out a release tag or branch instead; see the Monday section

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
source ${REPO_ROOT}/guides/env.sh      # defines GAIE_URL, ROUTER_STANDALONE_CHART, ROUTER_CHART_VERSION
export GUIDE_NAME="quickstart"
export NAMESPACE=llm-d-quickstart

# 1. Inference Extension CRDs (InferencePool lives here, not in llm-d)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

# 2. Namespace + the weights secret
kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -

# 3. The router: an Envoy proxy plus the Endpoint Picker, from an OCI Helm chart
helm install ${GUIDE_NAME} ${ROUTER_STANDALONE_CHART} \
  -f guides/recipes/router/base.values.yaml \
  -f guides/optimized-baseline/router/optimized-baseline.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

# 4. The model servers: vLLM pods the router discovers by label
kubectl apply -n ${NAMESPACE} -k guides/optimized-baseline/modelserver/gpu/vllm/base/

# 5. Send a request to the router service, not to a pod
export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')
# then, from a pod inside the cluster:
#   curl -X POST http://${IP}/v1/completions -H 'Content-Type: application/json' \
#     -d '{"model": "Qwen/Qwen3-32B", "prompt": "How are you today?"}'

The quickstart page, as I read it, says the default is 8 replicas of Qwen3-32B, while the guide page I read separately describes the default as 2 replicas at tensor parallelism 2, four GPUs in all, and the kustomize patch in the repository sets replicas to 2 with tensor-parallel-size 2. The docs disagree with each other, so size from the patch you actually apply, and adjust replicas and tensor parallelism if your cluster is smaller. Other accelerators have their own overlays under guides/optimized-baseline/modelserver/.

What the baseline router really decides

The interesting file is not the Helm command but the EndpointPickerConfig inside the values. The Optimized Baseline runs exactly two decision plugins, plus two producer plugins listed alongside them:

# guides/optimized-baseline/router/optimized-baseline.values.yaml (excerpt)
router:
  epp:
    pluginsConfigFile: "optimized-baseline-plugins.yaml"
    pluginsCustomConfig:
      optimized-baseline-plugins.yaml: |
        apiVersion: llm-d.ai/v1
        kind: EndpointPickerConfig
        # peakPrefillThroughput defaults to 15928, calibrated for Qwen3-32B on
        # H100 80 GB at TP=2. Other hardware or models: measure your own.
        plugins:
        - type: approx-prefix-cache-producer
        - type: inflight-load-producer
        - type: prefix-cache-affinity-filter
        - type: token-load-scorer
        schedulingProfiles:
        - name: default
          plugins:
          - pluginRef: prefix-cache-affinity-filter
          - pluginRef: token-load-scorer
  modelServers:
    matchLabels:
      llm-d.ai/guide: "optimized-baseline"   # the InferencePool selector

The prefix-cache-affinity-filter narrows candidates to endpoints likely to reuse the request's prefix, with a saturation-aware override that spreads load when endpoints run hot, and the token-load-scorer scores by total prefill token load per model server. The file's own comment is the honest part: the peakPrefillThroughput default of 15928 is calibrated for Qwen3-32B on H100 80 GB at TP=2, and anything else should measure its own value with the calibration recipe in the repository. Copying the default onto different hardware moves the point at which the router decides an endpoint is saturated.

Three releases, one direction

The release notes read as a sequence. Dates are as shown on the releases page, which gives no year.

The v0.10.0 notes state the theme as making the production path safe and boring, and then add that the tracking goal was not fully accomplished. The concrete items are small and operational: a rollouts guide, a high-availability preferred-backends guide for GKE, a graceful shutdown and request-draining ops guide, a KEDA startup-time mitigation through HPA stabilisation windows, and CI fixes so smoke tests reject HTTP errors. Only the inference server images are covered by cosign signing so far; the notes say the router and sidecar images are not included yet. Images from open pull requests go to quay.io and are intentionally unsigned.

What happens to a request when a pod dies

The most useful hardening artefact I found is the graceful-shutdown kustomize component, because it states a timeline and an invariant in comments. When a pod is deleted it leaves the InferencePool immediately while a preStop sleep gives the router and Service endpoints time to stop sending it traffic, and only then does vLLM start draining:

# recipes/modelserver/components/graceful-shutdown, as documented in the file:
#   t=0s    Pod is Terminating and leaves the InferencePool; preStop sleep starts
#   t=15s   SIGTERM. vLLM with --shutdown-timeout=45 rejects new requests,
#           lets in-flight ones finish
#   t=60s   Drain deadline: vLLM aborts anything still running
#   t=75s   SIGKILL if the process is still alive
#
# The invariant the file states, and the one that bites when you edit one number:
#   terminationGracePeriodSeconds >= preStop sleep (15)
#                                    + --shutdown-timeout (45)
#                                    + teardown buffer (15)
- op: add
  path: /spec/template/spec/terminationGracePeriodSeconds
  value: 75
- op: add
  path: /spec/template/spec/containers/0/lifecycle
  value:
    preStop:
      exec:
        command: ["/bin/sleep", "15"]

That invariant is a trap in waiting. The component sets the preStop sleep and terminationGracePeriodSeconds but deliberately not --shutdown-timeout, because each guide's patch replaces the whole args list and patches run after components. So the flag lives in the guide's vLLM args, and anyone raising it to protect longer generations must raise the grace period in the same change or Kubernetes sends SIGKILL mid-drain.

# recipes/router/base.values.yaml ships:
router:
  inferencePool:
    failureMode: "FailOpen"   # EPP down -> the gateway picks any endpoint

# The Inference Extension API reference says an unset failureMode means
# "FailClose": the parent drops the request when the Endpoint Picker fails.
# Wrong: assuming the router fails the way your last gateway did.
# Right: read the rendered InferencePool and decide which failure you prefer.

The second failure behaviour is the router itself. The Inference Extension API reference documents failureMode on the endpoint picker reference with two values and says an unset value means FailClose, where the gateway drops the request when the picker fails. llm-d's base router values set FailOpen, where the gateway forwards the request to an endpoint of its own picking. Neither is wrong. FailOpen keeps answering when the EPP is down but loses cache-aware and load-aware routing exactly when you are already in trouble; FailClose protects the backends from blind routing but turns an EPP outage into a full outage.

Trade-offs I would write down before adopting it

Nothing here is a reason to avoid llm-d, but several things in the files argue for caution before putting it in front of production traffic:

• Sandbox maturity. The CNCF page lists llm-d at Sandbox, accepted 12 March 2026. That signals an active project, not a stability guarantee.
• Churn in the release notes. v0.10.0 deprecates the WVA autoscaler guides, the llm-d-cuda and llm-d-aws images and the llmd-fs-connector, and moves llm-d-kv-cache into llm-d-router. Tracking main means absorbing renames.
• Unpinned defaults. In guides/env.sh the Inference Extension release defaults to latest, the router chart version to v0 and the endpoint-picker image tag to main.
• Hardware-specific tuning. The default scheduling constants are calibrated for one reference setup, and I found no figure for how much they matter elsewhere.

I also found no throughput or latency numbers in the sources I read that I could responsibly repeat, so I am not repeating any. The docs describe well-lit paths as tested, benchmarked recipes, and the v0.9.0 notes mention updated benchmarks for the optimized baseline; if the routing gain matters to your decision, measure it on your own model and traffic mix.

What to do on Monday

If you run vLLM on Kubernetes today, the cheapest useful experiment is the Optimized Baseline in a scratch namespace, treated as a rollout drill rather than a performance test:

1. Check out a release tag or release branch of the repository instead of main, and pin the three version variables below before installing.
2. Install the Optimized Baseline, then send ten requests sharing a long common prefix and read the vllm prefix-cache counters from each pod, as the guide's verification step suggests.
3. Delete a model server pod under load and watch for failed requests; compare against the drain timeline above.
4. Delete the EPP pod under load and record what your chosen failureMode does to latency and errors.
5. Write down which of the deprecated components in your cluster, if any, you would have to migrate.

# guides/env.sh honours these, but defaults every one to a moving target:
#   GAIE_VERSION         defaults to "latest"
#   ROUTER_CHART_VERSION defaults to "v0"
#   ROUTER_EPP_VERSION   defaults to "main"
# Pin them to what you tested, in the shell that runs the install.
export GAIE_VERSION=<the Inference Extension release you tested>
export ROUTER_CHART_VERSION=<the router chart version you tested>
export ROUTER_EPP_VERSION=<the endpoint-picker image tag you tested>
source guides/env.sh
echo "CRDs from: ${GAIE_URL}"        # should no longer say releases/latest

# After install, confirm what the cluster actually holds.
kubectl get inferencepool -n "$NAMESPACE" -o yaml | grep -n failureMode
kubectl get deploy -n "$NAMESPACE" -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.spec.template.spec.terminationGracePeriodSeconds}{"\n"}{end}'

The commands below pin the versions and read back what the cluster actually holds. They are my own checks around the repository's variables, not part of the official guide.

llm-d is worth evaluating if your cost is dominated by repeated prompt prefixes, because it moves the routing decision to where the cache state is known. Treat v0.10 as the release that wrote the failure behaviour down, not the one that proved it: pin versions, choose FailOpen or FailClose deliberately, keep the grace period above the drain timeout, and measure on your own traffic.