NVIDIA Dynamo on Kubernetes: Rust EPP and v1beta1 Upgrade

The sentence that sent me into the Dynamo documentation was short: the Rust Endpoint Picker replaces the Go EPP. If you run Dynamo behind a Kubernetes Gateway, that is the component that decides which GPU worker gets each request, and it just changed language, image and contract in one release. A change like that either costs you an afternoon or an outage, depending on whether you read the fine print first.

This is a close reading of the Dynamo 1.5.0 release notes, the deprecations ledger, the 1.0 announcement and the GAIE how-to, written from the operator's chair. I have not run this upgrade on a cluster, so there are no benchmarks of mine in here. Every number and flag comes from a page I read on 17 October 2026, and where the docs contradict themselves I say so.

Six releases in seven months: what an operator absorbed

Dynamo 1.0 was announced on 16 March 2026 and the latest release as I write is 1.5.1, from 6 October. The deprecations ledger shows how much moved in between. For anyone who deploys the platform rather than writes against it, the interesting lines are not the features but the removals, and 1.5.0 has the densest cluster of them so far.

Read that column top to bottom and a pattern appears: the project is paying down its own history on a monthly cadence. The release notes call 1.5.0 the eighteenth feature release, with 658 merged pull requests from 123 contributors. If you pin to a minor version and upgrade once a quarter, you will meet several of these removals in a single window.

What 1.0 promised, and the conditions attached

The 1.0 announcement leads with an eye-catching figure: up to 7x more requests served on NVIDIA Blackwell in SemiAnalysis InferenceX. The conditions are narrow and worth repeating. The run was DeepSeek R1-0528 in FP4, with 1k input and 1k output tokens, at roughly 50 tokens per second per user, and NVIDIA attributes the gain to disaggregated serving with wide expert parallelism on GB200 NVL72. A second headline, model loading up to 7x faster through ModelExpress, was measured on large mixture-of-experts models such as DeepSeek v3 on H200.

These are vendor-reported results on specific hardware, and I would not plan capacity from them. What I do take from the announcement is the shape of the operator-facing product: a Dynamo Graph Deployment Request that turns a model, target hardware and traffic goals written in YAML into a deployable graph, by combining the Planner with AIConfigurator.

That second piece has already changed. In 1.5.0 AIConfigurator became AISimulate and ships as its own package, at version 0.12.0, with aisimulate predict and aisimulate recommend commands. The legacy python -m dynamo.replay command was removed. If a runbook of yours still says AIConfigurator, it is now describing a name, not a tool.

The Rust Endpoint Picker and where it sits

Dynamo documents two ways to route on Kubernetes. In Frontend routing, the Dynamo Frontend tokenises the request, scores workers and forwards it; the docs recommend this by default. In Gateway API routing, a Gateway receives the request and asks the Endpoint Picker Plugin to choose a worker through the Gateway API Inference Extension, and the worker's Frontend sidecar then forwards in direct mode. The EPP, in the docs' own words, is not a public inference endpoint and does not run the model. You only meet it if traffic must enter through a Gateway for authentication, rate limiting or policy.

In 1.5.0 the Go implementation is gone and the Rust one is the default. It ships inside the Dynamo Frontend image at /epp, with a standalone image planned for a later release. Replicas synchronise over a fixed port set by DYN_EPP_REPLICA_SYNC_PORT, default 9092. On SIGTERM it marks its gRPC health service NOT_SERVING before draining, and it validates that Envoy sets response_body_mode to FULL_DUPLEX_STREAMED, a requirement worth checking in your gateway configuration. A fix in the same release stopped the KV-router scorer dropping the request body, which had caused 503 errors on multi-turn tool-calling requests.

The manifest below is assembled from the GAIE how-to for 1.5.1. The three details that matter are the epp component with no eppConfig, the sidecar in direct mode, and the block size agreeing everywhere.

# Rust EPP: one component of type epp, and NO eppConfig.
# eppConfig (even an empty object) selects the legacy Go contract
# and admission rejects it alongside a 1.5+ image.
spec:
  components:
  - name: Epp
    type: epp
    replicas: 1
    podTemplate:
      spec:
        containers:
        - name: main
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.5.1
          envFrom:
          - secretRef:
              name: hf-token-secret
          env:
          - name: DYN_MODEL_NAME
            value: Qwen/Qwen3-0.6B
          - name: DYN_KV_CACHE_BLOCK_SIZE
            value: "16"          # must equal the backend --block-size

  # Each routable worker gets a sidecar in direct mode: the EPP has already
  # chosen the worker, so the sidecar must not choose again.
  - name: VllmDecodeWorker
    type: decode
    frontendSidecar: sidecar-frontend
    podTemplate:
      spec:
        containers:
        - name: main
          # existing worker; publish KV events so routing sees real cache state:
          #   --enable-prefix-caching --block-size 16
          #   --kv-events-config '{"enable_kv_cache_events":true}'
        - name: sidecar-frontend
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
          args: ["-m", "dynamo.frontend", "--router-mode", "direct"]

Migrating from the Go EPP: blue/green or in place

The first thing the migration guide establishes is what you do not have to do. Upgrading only the Operator needs no EPP migration: the 1.5 Operator supports existing and new legacy DGDs while eppConfig stays set and the EPP image stays pinned to the 1.4 line. Removing eppConfig and moving to a 1.5-or-later image must happen as one intentional update, because admission rejects either mixed combination. An empty eppConfig object still selects the Go contract. A DGD also allows only one epp component, so you cannot run the two pickers side by side.

The docs prefer blue/green whenever you can afford a second DGD's worth of capacity, and the reason is in the table: it is the only path where the verification happens before production sees the change. The core of it is a canary HTTPRoute that adds an exact header match, so that only requests carrying the header reach the new pool, followed by a single JSON patch on the production route.

# Blue/green cutover, condensed from the docs' ten steps.
# Canary route: only requests carrying BOTH headers reach the new pool.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: qwen-rust-epp-canary
spec:
  parentRefs:
  - name: inference-gateway
  rules:
  - matches:
    - headers:
      - name: X-Gateway-Model-Name
        type: Exact
        value: Qwen/Qwen3-0.6B
      - name: X-Dynamo-EPP-Canary
        type: Exact
        value: rust-epp
      path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: qwen-rust-epp-pool
      port: 8000
    timeouts:
      request: 300s

# Only after Accepted and ResolvedRefs are True and a canary request succeeded:
# one patch moves production. Reverting it is the rollback.
kubectl patch httproute qwen -n my-model --type='json' -p='[
  {"op":"replace","path":"/spec/rules/0/backendRefs/0/name","value":"qwen-rust-epp-pool"}
]'

On a GPU fleet the capacity row is the real cost. Two complete worker sets for the overlap window can mean a second set of accelerators, so the choice is between spending that and accepting an unverified roll of the one component that every request passes through.

v1beta1 CRDs: storage, webhooks and the fields that became immutable

In 1.5.0 the DGD, DCD, DGDR and the DGDR scaling adapter moved to v1beta1 storage while still serving v1alpha1. The migration story is one line: upgrade the Operator so its built-in CRD migrator converts the stored objects. The v1alpha1 admission webhook endpoints are gone, so if you maintain your own Validating or MutatingWebhookConfiguration that the Helm chart does not manage, point it at the v1beta1 DGD paths with apiVersions set to v1beta1. The v1alpha1 DGDR API itself is not removed in this release.

Three smaller changes will catch manifests you did not know were fragile. The v1alpha1 componentType and v1beta1 type fields are immutable once set, enforced by CEL rules; legacy objects without a type may set it once. DGD and DCD components gained a typed roles list for multinode, with existing implicit manifests still valid. And runtimeVersionOverride on DGD, DCD and DGDR declares runtime compatibility when an image tag lacks a version, which matters if you mirror images under your own tags.

Also new: profiler output now generates v1beta1 spec.components with podTemplate, and a DGDR can set features.kvRouter.enabled to put DYN_ROUTER_MODE=kv on the frontend. Conversion fidelity fixes for the metadata envelope and service annotations landed in the same release, which is a polite way of saying the first conversions were lossy in places. Diff your objects before and after.

TLS and mTLS for the transports, and what is not covered yet

Release 1.5.0 adds opt-in TLS to the TCP request and response dispatch planes and matching environment variables and tls:// URL support for NATS, with opt-in mutual TLS adding client certificate verification on top. The Operator helps: ten certificate path fields in its InfrastructureConfiguration are injected as NATS and TCP TLS environment variables into every DGD pod, and the Operator validates the dependency chain at startup. That startup check is the useful part, since a half-configured chain then fails loudly at install time instead of mid-request.

The release notes also say the transport encryption extends to the Frontend HTTP server with mutual TLS, but under a Looking Ahead heading, so treat the client-facing hop as not covered in 1.5. Today you would keep terminating client TLS at the Gateway. I could not find a worked end-to-end example with certificate manifests in the pages I read, so I am not going to invent one.

KVBM deprecation, known issues and the traps

KVBM, the KV Block Manager that tiers cache across GPU, CPU, SSD and remote storage, is deprecated in 1.5.0 with removal targeted for 1.7.0. It still works, and the kvbm wheel and vLLM launch examples are unchanged. The ledger's migration advice is to use the engine's native KV offloading for host and disk tiering, and it is explicit that cross-node cache sharing belongs to a separate, early-stage project called KVCR which does not replace KVBM. The 1.0 announcement sold KVBM's object-storage support, so anyone who adopted that now has a clock running.

Then there is what is broken right now. Per the release notes, every inference request sent to the SGLang sidecar still fails with HTTP 500 in 1.5.1, a protocol mismatch with the bundled SGLang fixed upstream in v0.5.19, with the Dynamo fix targeted for 1.6.0 and no workaround. Some H.264 videos can also crash a worker when PyNvVideoCodec 2.2.3 decodes the final frame, and that affects 1.5.0 as well. Image and audio inputs are unaffected.

Two smaller traps. In an Istio-injected namespace, the Gateway proxy must not have an istio-proxy sidecar or you get HTTP 500 through the Gateway. And if prefix overlap seems to be ignored, the docs point at workers that are not publishing KV events or a block size that does not match. The readiness checklist is the DGD Ready condition, the generated InferencePool, the route conditions Accepted and ResolvedRefs both True, and EPP logs showing endpoint selection.

What to do on Monday

Start with an audit that changes nothing, then choose a migration path. In this order:

1. Find every DGD that still carries the Go contract by searching for eppConfig, and note which EPP image tag each pins.
2. Confirm the CRD stored versions after the Operator upgrade and re-point any self-managed webhook configuration at the v1beta1 paths.
3. Search your manifests and launch scripts for kvbm and decide whether engine-native offloading covers your tiering before 1.7.0.
4. If you run multimodal workers on 1.5.1, set DYN_MM_TRUST_EGRESS_PROXY=1 only where media genuinely goes through a proxy, and set DYN_MM_LOCAL_PATH for local input files.
5. Rehearse the blue/green cutover in a staging namespace with a canary header, including the rollback patch, before touching production.

The commands below are the audit and the cutover. The CRD name is whatever the grep finds on your cluster; I did not verify the plural form, so I look it up rather than assume it.

# 1. Which DGDs still carry the Go EPP contract, and what image do they pin?
kubectl get dynamographdeployment -A -o yaml | grep -n -E "eppConfig|dynamo-frontend:"

# 2. Did the Operator's migrator move storage to v1beta1? Look the CRD up first.
kubectl get crd | grep nvidia.com
kubectl get crd <name-from-above> -o jsonpath='{.status.storedVersions}'

# 3. Anything of yours still depending on KVBM before the 1.7.0 removal?
grep -rn -i kvbm deploy/ scripts/ charts/

# 4. Is the Gateway proxy free of an istio-proxy sidecar?
kubectl get pods -n my-model -l gateway.networking.k8s.io/gateway-name=inference-gateway \
  -o jsonpath='{.items[*].spec.containers[*].name}'

The rule I take from Dynamo 1.5 is to separate the upgrades that happen to you from the ones you choose. The Operator upgrade is safe to take on its own and migrates your CRDs; the EPP change is a choice with a rollback boundary, so schedule it, canary it and keep the old pool until the window closes. Treat every vendor benchmark as an advertisement and every removal date as a deadline.