KServe LLMInferenceService Guide: llm-d, WVA and vLLM

I went looking for a short answer to a plain question: if you already run vLLM on Kubernetes with a Deployment and a Service, what does KServe's LLMInferenceService give you that is worth a migration? The release posts for v0.17 and v0.18 are dense lists of pull requests, and the samples in the repository do not all agree with each other.

This is a close reading, not a field report. I read the v0.17 and v0.18 release posts, the Go types for the CRD at the v0.18.0 tag, the sample manifests at that tag, the vLLM Kubernetes guide and the v0.21.0 release notes. I did not run any of this on a GPU cluster, and where a manifest below is my own edit, it says so.

What LLMInferenceService replaces, and what it does not

LLMInferenceService is a separate custom resource from InferenceService. The v0.17 post calls it GenAI-first, as opposed to InferenceService, which targets predictive models, and says it supports distributed inference, KV-cache-aware scheduling, disaggregated prefill and decode, Gateway API Inference Extension integration and token-based rate limiting. Its minimal example creates the Deployment, Service, Gateway, HTTPRoute, InferencePool, InferenceModel and the endpoint picker pod from one object.

Compare that with the plain route. The vLLM guide for Kubernetes shows a PVC, a Secret for the Hugging Face token, a Deployment, a ClusterIP Service mapping port 80 to container port 8000, an in-memory emptyDir at /dev/shm for tensor parallelism, and HTTP probes on /health. It also lists KServe, llm-d, LeaderWorkerSet, KubeRay and several other stacks as alternatives, which tells you where the maintainers expect the plain manifests to stop.

The honest reading of the table is that you are trading stable, boring objects for a controller that does a great deal for you. If you run one model on a few GPUs behind a Service, the left column is enough and the right column is overhead. The right column starts to pay when routing, autoscaling or multi-node placement is the thing you keep rebuilding.

What v0.17 built: the llm-d pieces inside KServe

The v0.17 post, published on 2026-03-13, is the one that made the resource production-ready. Its contents, in the order I would care about them:

• KV-cache-aware scheduling: the endpoint picker tracks cache state across vLLM instances through ZMQ BlockStored and BlockRemoved events, and scores pods through an EndpointPickerConfig plugin pipeline instead of round-robin.
• Disaggregated prefill and decode: compute-heavy prefill and memory-heavy decode become separate workloads that scale separately, with KV blocks moved between pods through the NixlConnector over RDMA-based RoCE.
• Parallelism through LeaderWorkerSet: tensor, data, data-local and expert parallelism, configured with a worker spec.
• Envoy AI Gateway integration: an AIGatewayRoute defines llmRequestCosts for input, output and total tokens, and a BackendTrafficPolicy rate-limits per x-user-id header.
• Autoscaling with the Workload Variant Autoscaler (WVA), which watches KV-cache utilisation and queue depth through Prometheus and emits wva_desired_replicas for KEDA or HPA to act on.

The same release split the Helm chart from one into ten and says plainly that upgrading from v0.16 needs the upgrade guide rather than a plain helm upgrade. It reports 167 merged pull requests. The structural change matters more than the count: kserve, llmisvc and localmodel are now independent components, so you can install the generative stack without the predictive one.

The smallest manifest that does something

The sample below is the repository's own single-node GPU example at the v0.18.0 tag, trimmed, with my comments on the parts that surprised me. The empty router fields are the notable design choice: an empty object asks for the managed default rather than turning the component off.

# From docs/samples/llmisvc/single-node-gpu at the v0.18.0 tag (trimmed).
# One object. The controller derives the Deployment, Service, Gateway,
# HTTPRoute, InferencePool and the endpoint picker from it.
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: qwen2-7b-instruct-single
spec:
  model:
    uri: hf://Qwen/Qwen2.5-7B-Instruct   # where weights come from
    name: Qwen/Qwen2.5-7B-Instruct       # the name clients put in "model"
  replicas: 3
  router:
    scheduler: {}   # {} means "use the managed default", not "disabled"
    route: {}
    gateway: {}
  template:
    containers:
      - name: main
        resources:
          limits:   { cpu: "4", memory: 32Gi, nvidia.com/gpu: "1" }
          requests: { cpu: "2", memory: 16Gi, nvidia.com/gpu: "1" }
        livenessProbe:
          httpGet: { path: /health, port: 8000, scheme: HTTPS }  # HTTPS, not HTTP
          initialDelaySeconds: 120   # model load time, not a typo
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

The overview page lists the other top-level fields. spec.template is the single-node or decode workload, spec.worker is the multi-node worker pod that triggers a LeaderWorkerSet, spec.prefill is the prefill-only workload, and spec.parallelism carries tensor, data and expert sizes. The Go types add pipeline and dataLocal to that list.

Notice what is missing: no Service, no Deployment, no HTTPRoute. If you apply this and then list objects in the namespace, the comparison from the first section becomes concrete, because everything the raw route made you write shows up as controller output rather than your manifest.

Routing on KV-cache state, and the config you must keep in sync

The routing sample at v0.18.0 is where the design pays and where it bites. The scheduler config is an inline EndpointPickerConfig. Several values have to agree with the vLLM pods: the block size and hash seed, which the sample's comments call out, plus the hashing algorithm and ZMQ event endpoint that the sample sets on the vLLM side. If they drift, the scheduler's picture of which pod holds which prefix can be wrong, and it routes on that wrong picture.

# Based on precise-prefix-cache-routing/llm-inference-service-qwen2-7b-gpu-kv-cache-routing.yaml
# at v0.18.0. Values marked "must equal" have to match the vLLM pods.
spec:
  router:
    scheduler:
      config:
        inline:
          apiVersion: inference.networking.x-k8s.io/v1alpha1
          kind: EndpointPickerConfig
          plugins:
            - type: single-profile-handler
            - type: precise-prefix-cache-scorer
              parameters:
                tokenProcessorConfig:
                  blockSize: 64      # must equal vLLM --block-size (default 16)
                  hashSeed: "42"     # must equal PYTHONHASHSEED in the vLLM pods
                kvEventsConfig:
                  zmqEndpoint: "tcp://*:5557"   # vLLM publishes block events here
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
            - type: max-score-picker
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                # Wrong (as shipped in the sample): pluginRef: prefix-cache-scorer
                # Right: reference the plugin you actually declared above.
                - pluginRef: precise-prefix-cache-scorer
                  weight: 3
                - pluginRef: max-score-picker
  template:
    containers:
      - name: main
        env:
          - name: PYTHONHASHSEED
            value: "42"
          - name: VLLM_ADDITIONAL_ARGS   # the full sample also adds --kv-events-config (zmq publisher)
            value: "--prefix-caching-hash-algo sha256 --block-size 64"

The sample's comments state two of the constraints directly: blockSize must match vLLM's --block-size, whose default is 16, and hashSeed must match PYTHONHASHSEED in the vLLM pods. The manifest sets block size 64, hash seed 42 and sha256 prefix hashing on the vLLM side. The weights in the sample are queue-scorer 2, kv-cache-utilization-scorer 2 and a prefix scorer 3, so prefix affinity carries the highest weight.

The v0.17 post shows a different plugin vocabulary again: prefix-cache-scorer at weight 2.0, load-aware-scorer at weight 1.0 and a max-score-picker. The names changed between the blog and the samples, and v0.21.0 lists a change migrating the EndpointPickerConfig apiVersion. Scorer names are version-specific configuration, not a stable vocabulary.

Prefill and decode pools, multi-node and what v0.18 added

Disaggregation in this API is two pools in one object. The top-level template and replicas describe decode, and spec.prefill describes prefill with its own replica count and pod template. The sample moves KV blocks with the NixlConnector and asks for a RoCE network through a CNI annotation and an rdma/roce_gdr resource limit. That is real hardware: without an RDMA-capable network the pools can exist but the transfer path the sample assumes does not.

# Disaggregated prefill/decode, from single-node-gpu/...-pd-qwen2-7b-gpu.yaml
# at v0.18.0 (trimmed). Top level is the decode pool; spec.prefill is its own pool.
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: roce-p2   # RoCE network for the KV transfer
spec:
  replicas: 1          # decode
  prefill:
    replicas: 2        # prefill scales independently
    template:
      containers:
        - name: main
          env:
            - name: KSERVE_INFER_ROCE
              value: "true"
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom: { fieldRef: { fieldPath: status.podIP } }
            - name: VLLM_ADDITIONAL_ARGS
              value: "--kv_transfer_config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\"}'"
            - name: UCX_TLS
              value: "rc,sm,self,cuda_copy,cuda_ipc"
          resources:
            limits: { nvidia.com/gpu: "1", rdma/roce_gdr: 1 }

# Multi-node: spec.parallelism plus a worker pod spec becomes a LeaderWorkerSet.
# From the dp-ep sample: 16-way data parallel, 8 per node, expert parallel on.
spec:
  parallelism: { data: 16, dataLocal: 8, expert: true, tensor: 1 }

The v0.18 post, published on 2026-04-29, lists these additions that matter for this resource:

• Multi-node inference without Ray: a multinode/executor-backend annotation on the ServingRuntime selects ray, the default, or mp. In mp mode the node count comes from pipeline parallelism and GPUs per node from tensor parallelism, and the controller injects PIPELINE_PARALLEL_SIZE, TENSOR_PARALLEL_SIZE and WORKER_SVC into worker containers.
• LeaderWorkerSet as a scale target: LLMInferenceService can autoscale multi-node workloads through WVA v0.6.0, where earlier validation blocked combining spec.worker with spec.scaling. The release also upgrades llm-d to v0.6.
• The OpenAI Responses API: /v1/responses is now routed through the HTTPRoute.
• The restricted Pod Security Standards profile is enforced in the default template, and I would expect it to be the item that breaks an existing pod spec you carry over.
• A LocalModelNamespaceCache CRD for namespace-scoped model caches, vLLM upgraded to v0.19.0, and 130 merged pull requests from 27 contributors, 10 of them first-time.

One boundary to respect: the Ray-free annotation is described against the ServingRuntime, so it appears to belong to the InferenceService multi-node path. For LLMInferenceService the multi-node mechanism in the same post is the LeaderWorkerSet. I would not assume one flag covers both.

Autoscaling with WVA, read from the schema

The Go types at v0.18.0 are the most precise description of autoscaling I found, and they carry the validation rules as comments. The shapes below come straight from them. The numbers in my block are legal values, not tuned ones, and I did not measure scale-up latency for any model.

# Field names and rules read from the v0.18.0 Go types
# (pkg/apis/serving/v1alpha1/llm_inference_service_types.go).
# The numbers are mine, chosen to be legal, not tuned.
spec:
  # replicas and scaling are mutually exclusive (CEL rule on the workload).
  scaling:
    minReplicas: 2        # defaults to 1, minimum 1
    maxReplicas: 8
    wva:                  # required whenever scaling is set
      variantCost: "10.0" # default; relative cost per replica of this variant
      keda:               # exactly one of keda or hpa
        pollingInterval: 15          # seconds
        cooldownPeriod: 300          # seconds after the last active trigger
        initialCooldownPeriod: 600   # let the model load before KEDA judges it
        idleReplicaCount: 1          # must be below minReplicas, and minimum is 1
  # prefill.scaling takes the same shape, so each pool scales on its own.

Three rules fall out of the CEL validations. replicas and scaling are mutually exclusive on a workload. wva is required whenever scaling is set, and exactly one of hpa or keda must sit under it. And idleReplicaCount must be less than minReplicas while both have a minimum of 1, so using idleReplicaCount means minReplicas of at least 2.

That last rule contradicts the v0.17 post, which says the KEDA actuator supports idle scale-to-zero through idleReplicaCount. At the v0.18.0 tag the field has a minimum of 1, so it is an idle floor above zero. The v0.17 claim may describe an intent or a different version; the CRD at the tag is what the admission webhook enforces, so I trust that. Per the types, the controller also creates a VariantAutoscaling resource and a ServiceMonitor alongside the actuator.

Where it breaks: alpha APIs and moving parts

Everything above can be correct and still cost you a week, because the surface is moving. The evidence is in the releases themselves.

• The API version moved. The samples at v0.18.0 use serving.kserve.io/v1alpha1; the v0.21.0 notes include a change updating llmisvc samples from v1alpha1 to v1alpha2. A manifest from a blog post may not apply to your cluster.
• The autoscaling wiring moved. The v0.18.0 types describe a VariantAutoscaling resource created by the controller; v0.21.0 lists migrating WVA from the VA CRD to annotation-based discovery. Dashboards or policies that referenced the old object would break.
• The tokenizer path moved. v0.21.0 lists replacing the UDS tokenizer sidecar with a vLLM render deployment. Anything that scraped or patched the sidecar needs a second look on upgrade.
• The install moved. v0.17 split one Helm chart into ten and requires the upgrade guide from v0.16, and v0.21.0 lists restructuring CRD management for independent installation.

The newest release I could confirm is v0.21.0, published on 2026-09-25, with v0.19.0 on 2026-06-14 and v0.20.0 on 2026-08-06 between it and the v0.18 post. So a guide built on v0.18 is three minor versions old. I checked the tags and the release notes, but I did not read the v0.19 and v0.20 notes in full, so I make no claim about what they changed.

What to do on Monday

Treat this as an evaluation, not a migration. The order below is cheapest-first and each step either ends the evaluation early or earns the next one.

1. Decide whether you need it. One model, a handful of replicas, no multi-node: stay on the vLLM Deployment. Needing KV-cache-aware routing, a prefill and decode split, multi-node or autoscaling on inference metrics is the signal to continue.
2. Pin the version and read the CRD your cluster serves, using the commands below, before copying any YAML from a post, including this one.
3. Start with the single-node sample and an empty router. Confirm the controller generates the objects you expected before you touch scheduler config.
4. Add routing config only with a measurement in hand: your own prefix-sharing workload, the block size, hash seed and hashing algorithm set identically on both sides, and the scheduler logs open while you test.
5. Add scaling last, with initialCooldownPeriod above your model load time and minReplicas of at least 2 if you want an idle floor. Re-run the dry-run apply after every KServe upgrade.

# 1. Which release is actually installed? Do not trust the blog post's version.
kubectl get deploy -A -o wide | grep -i kserve

# 2. Which API version does the CRD serve here? v1alpha1 on v0.18, v1alpha2 in newer samples.
kubectl get crd llminferenceservices.serving.kserve.io \
  -o jsonpath='{.spec.versions[*].name}{"\n"}'

# 3. Read the schema your cluster enforces, not the one you remember.
kubectl explain llminferenceservice.spec.scaling --recursive

# 4. Let the admission webhook judge the manifest before anything is created.
kubectl apply --dry-run=server -f my-llmisvc.yaml

# 5. After applying, see what the controller generated from your one object.
kubectl get llminferenceservice,deploy,svc,httproute,inferencepool -n my-ns

The commands are plain kubectl. kubectl explain reads the schema from the installed CRD, which makes it the one source that cannot be out of date for your cluster.

My rule after reading all of it: adopt LLMInferenceService for the routing, disaggregation and autoscaling you would otherwise build, keep a plain vLLM Deployment for everything simpler, and treat every YAML snippet, including mine, as a claim about one version that your own CRD has to confirm.