Gateway API Inference Extension: EPP Moves to llm-d, What Now

The scouting note that started this post said the Endpoint Picker move was announced with v1.6.2 on 17 September. That is not what the sources say. The proposal issue was closed as completed on 13 May, the v1.6.0 release notes already list the move, and v1.6.2 is a patch release. Getting the date wrong matters, because it changes how long your cluster has been running code that its upstream repository no longer accepts changes to.

This is a migration note built from what I read: the SIG repository README, the proposal issue, the v1.6.0 and v1.6.2 release notes, the InferencePool API page, and the llm-d router and payload processor repositories, including the Helm chart README and operations guide. Where a source was silent I say so rather than fill the gap. I did not run any of these charts myself.

What moved, what stayed, and where each piece lives now

The README of the SIG repository states that the Endpoint Picker (EPP), the InferenceObjective and InferenceModelRewrite APIs and the Body Based Router (BBR) packages have moved. EPP and its APIs went to llm-d/llm-d-router, and BBR went to llm-d/llm-d-inference-payload-processor. It adds that no new code will be accepted to those packages in the SIG repository and that they will be archived soon.

The practical reading: if you ever deployed the upstream EPP to do real scheduling, your production dependency is now a different project, with a different release cadence and different maintainers. If you only needed the InferencePool resource and a conformance-grade picker, nothing changed under you.

The timeline, corrected: May for the move, August for the release

The proposal issue (number 2430) was opened on 26 February 2026. Two maintainers approved on 9 March, the title was widened to include BBR on 12 March, an execution plan was shared on 11 April, and the issue closed on 13 May with the maintainer writing that the move happened. The v1.6.0 notes, dated 17 August on the releases page, are where the removal of InferenceObjective, InferenceModelRewrite and EndpointPickerConfig from this repository is recorded.

The v1.6.2 release of 17 September pins the conformance manifest image to the new tag and lists three fixes: body responses in FULL_DUPLEX_STREAMED mode carrying proper BodyMutation streamed responses so gateway streams are not dropped (number 3028), request headers sent before the body in ext_proc gRPC streams to stop spurious HTTP 500s with chunked bodies (3033), and the buildx TARGETARCH fix for exec format errors on arm64 (3032). Those are good reasons to be on 1.6.2, but they are not the move.

What stays upstream: the InferencePool contract

The thing your gateway integration depends on is the InferencePool resource, and that remains in the SIG project under the inference.networking.k8s.io/v1 group. The API page documents it with a selector that must exactly match your model server pod labels, targetPorts, and an endpointPickerRef pointing at the EPP service. As of v1.5.0 that reference is optional, and the v1.6.0 notes repeat the change.

# InferencePool as shown on the API page: still upstream, still the contract
# between your Gateway and whichever EPP you run.
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: vllm-qwen3-32b
spec:
  selector:
    matchLabels:
      app: vllm-qwen3-32b      # must exactly match the model server Pod labels
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: vllm-qwen3-32b-epp   # the EPP Service, now typically from llm-d charts
    port:
      number: 9002
    failureMode: FailOpen      # gateway picks an endpoint itself if EPP does not answer

So the re-pointing question is narrow. The pool and the route stay as they are; what changes is which chart and image produce the Service that endpointPickerRef names. The docs site says this directly: production users should implement their own EPP or use an existing one, and it gives llm-d-router as the example.

Re-pointing the Helm install: legacy keys now fail the install

The llm-d router ships three charts under config/charts: llm-d-router-gateway for Gateway mode, llm-d-router-standalone for a self-managed proxy, and routerlib, a library chart that cannot be deployed on its own. Its README says the new charts fail installation when they find legacy top-level keys or legacy command-line flags. That is helpful, because a silent half-migration would be worse, but it means your old values file will not carry over.

Gateway mode is the production setup the README recommends: the EPP backs an InferencePool that an HTTPRoute references on a shared Gateway. The chart README prerequisite is to install the Gateway API Inference Extension CRDs first, which is a useful reminder that the CRDs still come from the SIG project.

# values-llm-d.yaml - keys taken from the llm-d-router chart README.
# modelServers.matchLabels is required and defaults to empty.
router:
  modelServers:
    matchLabels:
      app: vllm-qwen3-32b      # same label your InferencePool selector used
    type: vllm                 # default; sglang and triton variants exist
    targetPorts:
      - number: 8000
  inferencePool:
    create: true
    failureMode: FailOpen      # chart default
  epp:
    replicas: 1

# Gateway mode install, from the chart README (path is local to the repo checkout).
# Wrong: helm upgrade with your old values file - legacy keys abort the install.
# Right: translate keys first, then install the gateway chart.
helm install vllm-qwen3-32b ./config/charts/llm-d-router-gateway \
  -f values-llm-d.yaml \
  --set provider.name=gke     # none, gke or istio

Note what the README does not give: an OCI registry path or Helm repository URL. Every documented command installs from a local checkout of config/charts. I would pin that checkout to a tag in CI rather than track main.

Body Based Router becomes the Inference Payload Processor

BBR moved to llm-d-inference-payload-processor, whose README describes the Inference Payload Processor (IPP) as a pluggable framework that inspects and mutates inference payloads as an ext-proc service beside the gateway proxy. Its main use is payload-aware routing: it reads signals such as the model name and injects headers so one Gateway endpoint can serve many models and LoRA adapters. IPP chooses the pool and the EPP chooses the pod within it.

Be careful with the word rename. The IPP README never mentions BBR, so I cannot confirm from sources that it is a drop-in replacement with the same configuration. What I can confirm is the deployment shape: its chart has three modes. Istio installs an EnvoyFilter, GKE installs a GCPRoutingExtension, and None deploys the core resources so you wire the proxy integration yourself. Read its chart README before assuming your BBR config translates.

Failure modes and high availability in the new router

The operations guide is where the move becomes a reliability question. In Gateway mode, router.inferencePool.failureMode defaults to FailOpen, so the Gateway routes straight to model servers if the EPP is unreachable. In standalone mode, router.proxy.failOpen defaults to true and unpicked requests go to an inference_pool_fallback cluster. Agentgateway has no fail-open setting and rejects requests while the EPP is unreachable.

Fail-open covers an unreachable EPP, not a draining one that returns an error response. And leader election has a cost: the guide says failover takes 10 to 30 seconds before a standby begins serving, with unoptimised routing in the gap. It recommends priority routing in standalone service mode for sub-second failover.

# HA examples from the llm-d router operations guide.

# Active-active: replicas share load, flow control is per replica.
router:
  epp:
    replicas: 3
    haMode: all-active
    crossReplica:
      enabled: true            # alpha Redis syncer; needs Redis 7.4+ you provision
      redis:
        address: redis.example:6379

---
# Active-passive with priority routing (standalone, service mode).
router:
  proxy:
    mode: service
    priorityRouting:
      enabled: true
      primaryReplicas: 1
      standbyReplicas: 1

The trade-offs and what can bite you

The governance argument is real and unresolved. In the proposal thread, a participant wrote that what llm-d does not provide that the upstream project did is neutral governance, and that moving code to one implementation was unprecedented. The thread ended with participants agreeing to disagree. The motivation on the other side was duplication: the EPP existed upstream and in the llm-d scheduler, and the advanced plugins were mostly developed in llm-d.

Three concrete risks follow from the sources. First, Envoy Gateway users: a comment on 26 May asked how they can keep using the EPP since llm-d does not list it as a supported implementation, and the thread shows no answer. Second, downstream vendors: the thread notes that Dynamo vendors the EPP code and adds plugins. Third, the router README says the only supported Envoy body modes are FULL_DUPLEX_STREAMED for both request and response bodies, so a proxy configured otherwise will not behave.

What to do on Monday

Treat this as an inventory task first and a migration second. In order:

1. Find which EPP you run: images, Helm releases and the chart your pool's endpointPickerRef Service comes from.
2. Decide whether you need real scheduling or only conformance. If only conformance, the lwepp image stays upstream; if scheduling, plan the llm-d router.
3. Translate your values file with the four-row key mapping, then install the new chart into a staging namespace beside the old one.
4. Verify the proxy uses FULL_DUPLEX_STREAMED bodies, and exercise a streaming completion end to end.
5. Move to v1.6.2 for the upstream components you keep, and stop applying fixes to the archived packages.

# 1. Which InferencePools exist, and which EPP Service does each name?
kubectl get crd | grep inference.networking
kubectl get inferencepools -A -o yaml | grep -A4 endpointPickerRef

# 2. Which images from the old repo are running?
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.spec.containers[*].image}{"\n"}{end}' \
  | grep gateway-api-inference-extension

# 3. Do your Helm values still use legacy top-level keys the new charts reject?
helm list -A
helm get values RELEASE -n NAMESPACE -o yaml \
  | grep -nE '^(inferenceExtension|inferencePool|inferenceObjectives|experimentalHttpRoute):'

An empty result in step 3 does not prove you are safe, since values can also come from a parent chart; treat it as a first pass.

The rule I take from this: separate the API contract from the implementation, and track them as different dependencies. The InferencePool contract stays with Kubernetes; the picker that does the scheduling now lives with llm-d, and your values file, images and failure behaviour have to follow it.