Kubernetes 1.37 DRA Extended Resources GA: Run example.com/gpu Without a Device Plugin

The question I wanted answered before anyone touched a GPU cluster was a boring one: if the vendor ships a DRA driver tomorrow, which of our manifests have to change? For years the honest answer to anything involving Dynamic Resource Allocation was all of them, because a device plugin hands out example.com/gpu as a node-level counter, while DRA hands out devices through ResourceClaim objects that the workload has to name.

Kubernetes 1.37 changes that answer. This post reads the v1.37 DRA release post of 3 September 2026, the KEP-5004 design document, the DRA concept, API and features pages, the device taints page, and KEP-5055 and KEP-6132, then turns them into a migration plan. I have not run this on a GPU cluster, so every behaviour below is the documentation's or the KEP's claim, attributed as such, and anything the sources do not settle is marked unconfirmed.

What 1.37 shipped for DRA, by stage

The headline is that DRA Extended Resource support is GA. The release post says the team had been building toward it for three straight releases, and the task page in the docs marks it Stable since Kubernetes v1.37 and enabled by default. The KEP's implementation history reads: accepted in 1.34, alpha in 1.35, beta in 1.36, GA in 1.37. The docs page says first available in v1.34, so the sources disagree by one release on the alpha. Treat 1.37 as the release that matters.

Three other features went stable in the same release, and a handful more moved stage. The table lists them with the gate names from the release post and the DRA pages; where a source gave no gate, the cell says so.

The release post also lists new or second-round alphas: derived attributes (CEL expressions that bridge differently named attributes across drivers), list-typed attributes, node allocatable resource requests, resource availability snapshots through ResourcePoolStatusRequest, and optional node operations. I leave those out below, because none of them touches the extended resource migration.

Why an unchanged manifest now works

The workload side does not change at all, and that is the design goal. KEP-5004 gives three motivations: existing applications run without modification, application developers move to DRA at their own pace, and cluster admins move at their own pace, possibly one node at a time. The only new object is a cluster-scoped DeviceClass that an admin creates. Below are the Pod and the class, with the class taken from the DRA API page.

# Workload side: unchanged. This is the request a device plugin satisfies today.
# Extended resources need requests and limits to match, as in the docs' dongle example.
apiVersion: v1
kind: Pod
metadata:
  name: demo
spec:
  containers:
  - name: demo
    image: registry.example.com/gpu-app:1.0
    resources:
      requests:
        example.com/gpu: 1
      limits:
        example.com/gpu: 1
---
# Admin side: one cluster-scoped object, taken from the DRA API page in the docs.
# extendedResourceName is the whole migration surface.
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type
        == 'gpu'
  extendedResourceName: example.com/gpu

Nothing in the Pod mentions DRA. The admin step is the entire migration surface: extendedResourceName tells the scheduler that devices matching the class's CEL selectors can satisfy requests for that name. The API reference says the name should be unique among classes. With two classes on one name, the one created later wins, and on a tie the lexicographically first name wins. The KEP presents that as a non-disruptive way to move a name between classes: create the new class, then clear the field on the old one.

There is a second route with no admin mapping at all. Any DeviceClass can be requested as the extended resource named deviceclass.resource.kubernetes.io/ followed by the class name, and the docs say the resulting ResourceClaim holds an ExactCount request for that many devices. The catch, per the KEP, is a naming mismatch: a DeviceClass name may be a DNS subdomain of up to 253 characters, while the name part of an extended resource is limited to 63, so a class only works as an implicit name if it fits both rules. Otherwise set extendedResourceName explicitly.

# Implicit form: no extendedResourceName on the class and no ResourceClaim in the Pod.
# The name is the fixed prefix plus the DeviceClass name (here gpu.example.com).
resources:
  limits:
    deviceclass.resource.kubernetes.io/gpu.example.com: 2

Read the last two rows twice. The same name on two different nodes is fine and the same name on one node is not, which is what makes a gradual migration possible. On quota, the KEP specifies adjusted accounting so that a ResourceQuota written for the old world keeps meaning something. I did not verify how that shipped.

What the scheduler does behind your back

When a Pod requests a DRA-backed extended resource, the scheduler builds a ResourceClaim the Pod's author never wrote. The KEP is precise about it. The claim is namespaced and owned by the Pod, created by the scheduler's dynamicresources plugin in PreBind (an in-memory copy exists from Reserve), and there is at most one per Pod however many extended resources it requests. It carries the annotation resource.kubernetes.io/extended-resource-claim naming the Pod, and each request is named after the container and the resource, such as container-0-request-2 for the first container's third DRA-backed resource. The KEP says that naming is informational and the kubelet must not rely on it.

The kubelet finds the claim through a new Pod status field, extendedResourceClaimStatus, whose requestMapping ties each container and extended resource name to a request in the claim. It then asks the DRA driver to prepare the devices and passes them into the matching containers through CDI. The KEP says containers can be init or regular containers but not ephemeral ones, which is worth remembering before you reach for a debug container on a GPU Pod.

# 1. Did the scheduler create a claim for this Pod? The mapping is in Pod status.
kubectl get pod demo -n default -o jsonpath='{.status.extendedResourceClaimStatus}'

# Shape the KEP gives for that field (claim name will differ on your cluster):
#   status:
#     extendedResourceClaimStatus:
#       resourceClaimName: ccc-gpu-57999b9c4c-vpq68-gpu-8s27z
#       requestMapping:
#       - containerName: container-name
#         extendedResourceName: foo.domain/bar
#         requestName: container-0-request-2

# 2. Read the claim itself. The annotation resource.kubernetes.io/extended-resource-claim
#    marks a claim the scheduler wrote, as opposed to one a human wrote.
kubectl get resourceclaim ccc-gpu-57999b9c4c-vpq68-gpu-8s27z -n default -o yaml

# 3. Signals the KEP names. Confirm each exists on YOUR /metrics before alerting on it.
#    scheduler_resourceclaim_creates_total{status="failure"}
#    resourceclaim_controller_resource_claims{source="extended-resource"}
#    scheduler_pending_pods{queue="unschedulable"}

Those fields are the first places to look when something is wrong. The KEP also names the signals: scheduler_pending_pods rising, kubelet_started_containers_errors_total, a new source label on resourceclaim_controller_resource_claims, and a new scheduler_resourceclaim_creates_total counter with a status label. The KEP calls these SLIs, so confirm each one exists on your own /metrics before you build an alert on it.

Migrating a cluster one node at a time

The KEP is explicit that this is the intended path: device plugin drivers can be replaced by DRA drivers per node. It also names the signal that a swap did not finish. If node.status.capacity for the extended resource does not fall to zero, the device plugin was not properly replaced. The constraint that sets the order is the one from the comparison table, that a node cannot advertise one name from both backends.

# One node at a time. Steps 2 and 3 are the vendor's procedure, not Kubernetes'.
NODE=gpu-worker-01

# 1. Stop new Pods landing here. Draining is my choice, not a KEP requirement:
#    it avoids a Pod holding a device-plugin allocation while the backend swaps.
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data

# 2. Remove the device plugin from this node (vendor-specific).
# 3. Install the DRA driver on this node (vendor-specific).

# 4. The KEP's signal that the old backend is really gone: capacity must fall to zero.
#    Expect <none> in the GPU column. Anything else means the plugin is still advertising.
kubectl get node "$NODE" \
  -o custom-columns='NAME:.metadata.name,GPU:.status.capacity.example\.com/gpu'

# 5. The new backend must be advertising instead: ResourceSlices for this node.
kubectl get resourceslices \
  -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,DRIVER:.spec.driver'

# 6. The DeviceClass from earlier must exist BEFORE a Pod can use DRA here
#    (an application can only request devices from DRA after the class is created).
kubectl get deviceclass gpu.example.com

# 7. Only now let Pods back in, and run the unchanged workload as the test.
kubectl uncordon "$NODE"

The KEP's rollback paragraph is where the cost hides. Disabling the gate is safe for running Pods, which keep working and survive restarts because the CDI devices already prepared for them do not change. Re-enabling is not symmetrical, as the warning below says.

Make the gate consistent before the first Pod depends on it. The KEP's rollout risk is a Pod using the feature before kube-apiserver, scheduler, controller-manager and kubelet all support it, which can fail scheduling or fail to start the Pod. On 1.37 the gate is on by default, so the practical form of that risk is version skew during an upgrade: finish the control plane and every kubelet before you create the first DeviceClass with extendedResourceName. That ordering is my inference from the KEP, not a sentence in it.

Device taints are stable, so maintenance gets a dry run

Device taints and tolerations are Stable since 1.37 and were first available in 1.33. A taint has a key, a value and an effect, and the effect applies to the ResourceClaim using the device and to every Pod referencing it. NoSchedule stops new allocations. NoExecute implies NoSchedule and evicts Pods already scheduled, through the device taint eviction controller in kube-controller-manager. None is ignored by both the scheduler and the eviction controller, so a driver can flag a degraded device with no consequences.

Admins can taint without touching the driver through DeviceTaintRule, stable since 1.37 and first available in 1.35. Each rule adds one taint to the devices its deviceSelector matches, and a rule with no selector matches nothing, which the docs describe as protection against evicting every Pod by leaving the selector out. The selector can name a DeviceClass, driver, pool and device. Drivers of node-local devices are encouraged to use the node name as the pool name, so a pool selector taints one whole node's devices. The page also documents a safe procedure, and the block below follows it.

# Step 1: a DRY RUN. effect None is ignored by the scheduler and the eviction controller.
# Adapted from the device taints page; dra.example.com is the docs' fictional driver.
apiVersion: resource.k8s.io/v1
kind: DeviceTaintRule
metadata:
  name: example
spec:
  deviceSelector:
    driver: dra.example.com      # no selector at all would match NO devices, by design
  taint:
    key: dra.example.com/unhealthy
    value: Broken
    effect: None

# Step 2: read what the rule WOULD do. For effect None the message reports published
# devices selected, allocated devices selected and Pods that would be evicted.
#   kubectl describe devicetaintrule example
# That message is written once. Delete and recreate the rule to refresh it.

# Step 3: when the numbers are what you expected, switch to real eviction.
#   kubectl patch devicetaintrule example --type merge \
#     -p '{"spec":{"taint":{"effect":"NoExecute"}}}'

# Step 4: wait for the eviction controller to finish.
#   kubectl wait --for=condition=EvictionInProgress=false DeviceTaintRule/example

Tolerations live on the ResourceClaim request rather than the Pod, so every Pod sharing a claim tolerates the same things, and the KEP rejected Pod-level tolerations for that reason. That leaves a gap I could not close from the sources. A Pod using only an extended resource has no ResourceClaim of its own, because the scheduler writes the claim. In the pages I read, neither the device taints page nor KEP-5004 says how such a Pod tolerates a taint, so test a tolerated-taint case before you rely on one. KEP-5055 also says a taint is not an access control mechanism, and that adding one to a live cluster is racy because the scheduler has to observe it first.

The scheduler throughput feature that was demoted

The release post's headline number is that PreQueueingHint cuts the requeue path from O(N squared) to O(1) and roughly doubles scheduling throughput in early benchmarks. KEP-6132 gives the mechanism and the measurement. Each ResourceClaim event used to trigger an evaluation of every unschedulable Pod, so N Pods and N claim events cost O(N squared) work. A Pod informer index now narrows each event to the Pods that reference the claim. On the SchedulingWithResourceClaimTemplate benchmark in test/integration/scheduler_perf, the KEP records about 297 Pods per second before and about 711 after, roughly 2.4 times.

It is also a clean example of what alpha means. The KEP says the feature was first aimed at beta in 1.37, then demoted to alpha with the default off, late in the cycle, after review found correctness bugs in the narrowing logic. The failure it guards against is a hint that wrongly says no Pod is affected, which would leave Pods unschedulable until a periodic flush. The KEP's implementation history records a fix and a beta promotion for v1.38. My advice is not to enable SchedulerPreQueueingHints on a production scheduler in 1.37 for a gain you have measured only on a burst benchmark. If you scale thousands of Pods against ResourceClaimTemplates, test it in staging first.

What breaks, and where to look

Every row below is a failure mode or constraint stated by the KEP or the docs. The third column is where I would look first.

Two of these are quiet. The duplicate-class case raises no error by design, since the KEP calls it a non-disruptive, non-error way to switch a name between classes, so a stray second class is honoured silently. And the KEP's own drawbacks section is one line: it adds complexity to the scheduler. Cluster autoscaler is the open edge. The KEP says the autoscaler may need to set the new DynamicResources field in its own node state and may need a public method for it, so read your autoscaler release notes before you scale GPU node groups on DRA-backed names.

What to do on Monday

None of this needs a GPU to start. The first steps are read-only or run on a test cluster, and they tell you whether the migration is worth scheduling.

1. Confirm the control plane and every kubelet are on 1.37, then read the Assign Extended Resources task page for your version.
2. List the extended resource names your Pods request and the device plugin behind each one. Each is a candidate extendedResourceName.
3. Ask each driver vendor whether a DRA driver exists for that hardware and which selectors its DeviceClass expects. I did not verify any vendor's driver.
4. Pick one non-critical node, migrate it with the sequence above, run the unchanged workload on it, and read status.extendedResourceClaimStatus on the Pod.
5. Write the rollback as a drain, and add a dry-run DeviceTaintRule to the maintenance runbook.

# Which extended resources do Pods in this cluster actually request? (read-only)
# Each name with a slash and a vendor-style domain is a candidate extendedResourceName.
kubectl get pods -A \
  -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.resources.requests}{"\n"}{end}{end}' \
  | grep -E '"[a-z0-9.-]+/[a-z0-9.-]+"' | sort | uniq -c | sort -rn

# Is the resource.k8s.io API served, and has any DRA driver published devices yet?
kubectl api-resources --api-group=resource.k8s.io
kubectl get deviceclasses
kubectl get resourceslices

# Any DeviceClass already claiming an extended resource name? (spot duplicates now)
kubectl get deviceclasses \
  -o custom-columns='NAME:.metadata.name,EXT:.spec.extendedResourceName,CREATED:.metadata.creationTimestamp'

If all of that turns out boring, you are done, which is the point of the feature: the manifests stay as they are and the backend moves underneath them.

The rule I take from this release is to migrate the allocation layer before rewriting a single manifest. Extended Resource support turns DRA adoption from an application rewrite into a per-node infrastructure change with a documented rollback, and the parts still in alpha, PreQueueingHint above all, are not the ones to build the plan on.