Kubernetes DRA GPU Fallback: Prioritized List in 1.36

The request I kept seeing in GPU clusters was never really for a model of card. It was for enough memory to hold the model, with a preference for the fast card and a tolerance for the slower one. The device plugin way of asking, a fixed count of one named resource on a container, cannot say that, so teams end up with two Deployments, two sets of manifests and a human deciding which one to apply.

Kubernetes 1.36 closes that gap with the prioritized list in Dynamic Resource Allocation (DRA), which went stable. This post is a reading, not a benchmark. I went through the Kubernetes DRA team's v1.36 post, the current concept and task pages, and the NVIDIA DRA driver documentation, and I did not run any of this on a GPU cluster. Every field name below comes from a page I read, and the places where I assembled a manifest myself are marked.

Why a claim instead of a count

DRA works like dynamic volume provisioning. Drivers publish ResourceSlices describing devices, admins or drivers define DeviceClasses as categories, and a workload asks for devices through a ResourceClaim or a ResourceClaimTemplate that filters with CEL expressions on device attributes. The docs list the advantages over device plugins as flexible CEL filtering, device sharing between containers and Pods, per-workload configuration and centralised categories, where device plugins need per-container requests and offer no sharing or expression filtering. The core feature has been stable since Kubernetes 1.35, and it first appeared in 1.30.

A claim is an object you can inspect, share and template, and that is what makes fallback possible. A count on a container is a number the scheduler either satisfies or does not. A claim is a small query, and a query can have a second choice. The cost is a new set of objects to learn: ResourceSlice, DeviceClass, ResourceClaim, ResourceClaimTemplate.

The prioritized list in one manifest

The Kubernetes post puts it in one sentence: give me an H100, but if none are available, fall back to an A100. In the API this is the firstAvailable field on a request, holding an ordered list of subrequests. The API concepts page marks it Stable since 1.36, enabled by default, and first available in 1.33. The scheduler selects the first subrequest that can be allocated, and each subrequest can carry its own selectors and its own count.

Below is a claim template in the shape of the docs example, adapted to the NVIDIA driver. The productName attribute and the gpu.nvidia.com DeviceClass come from the driver's ResourceSlice reference, which lists values such as NVIDIA A100-PCIE-40GB. The exact H100 product string is not on that page, so I use a substring match and you should confirm the real string with kubectl get resourceslices -o yaml on your own nodes.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-else-a100
spec:
  spec:
    devices:
      requests:
      - name: gpu
        # firstAvailable replaces "exactly": the scheduler walks this list
        # in order and takes the first entry it can allocate.
        firstAvailable:
        - name: h100
          deviceClassName: gpu.nvidia.com
          selectors:
          - cel:
              expression: |-
                device.attributes["gpu.nvidia.com"].productName.contains("H100")
        - name: a100-pair
          deviceClassName: gpu.nvidia.com
          count: 2   # a fallback may ask for a different shape, not just a different card
          selectors:
          - cel:
              expression: |-
                device.attributes["gpu.nvidia.com"].productName.contains("A100")

Note that the fallback here is not like for like: one H100 first, two A100s second. That is the point of letting each subrequest carry its own count. It is also the trap, because the workload that starts must be able to run on either shape, and the claim cannot make that true for you.

Wiring it to a Pod and reading the result

The Pod references the template in spec.resourceClaims and then names the claim in a container's resources.claims, the same two-step wiring as the task page's Job example. Using a ResourceClaimTemplate gives each Pod its own generated claim, whose lifetime is bound to the Pod. Use a hand-written ResourceClaim instead when several Pods must share the same devices.

apiVersion: v1
kind: Pod
metadata:
  name: inference
spec:
  containers:
  - name: server
    image: ubuntu:24.04
    command: ["sleep", "9999"]
    resources:
      claims:
      - name: accelerator          # must match the entry below
  resourceClaims:
  - name: accelerator
    resourceClaimTemplateName: h100-else-a100   # one generated claim per Pod

# Then, per the task page, drill down when it does not start:
#   kubectl describe pod inference
#   kubectl get resourceclaims
#   kubectl describe resourceclaim <generated-name>

The scheduler also uses the position of the chosen subrequest as one input when it scores nodes, so nodes that can satisfy a higher-ranked subrequest are more likely to be chosen. After scheduling, the docs' troubleshooting advice applies: drill down from the Job to the Pods to the ResourceClaims and run kubectl describe at each level, looking at status and events.

Partitionable devices: MIG without lying to the scheduler

Partitionable devices were beta in 1.36 and the current docs say Beta since 1.36, enabled by default, behind the DRAPartitionableDevices feature gate on the kube-apiserver and kube-scheduler. The model is counters. A ResourceSlice declares named counter sets describing what the physical device has, and each logical device declares what it consumes from them. In the docs' example, two devices each consume 6Gi from a shared counter holding 8Gi, so only one of them can be allocated at a time, and the consumer's ResourceClaim API does not change.

# From the DRA features page: two devices that each consume 6Gi of a shared
# 8Gi counter. Both are advertised, only one can ever be allocated.
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: resourceslice-with-countersets
spec:
  nodeName: worker-1
  pool: {name: pool, generation: 1, resourceSliceCount: 2}
  driver: dra.example.com
  sharedCounters:                 # counter sets live in their own slice...
  - name: gpu-1-counters
    counters:
      memory:
        value: 8Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: resourceslice-with-devices
spec:
  nodeName: worker-1
  pool: {name: pool, generation: 1, resourceSliceCount: 2}
  driver: dra.example.com
  devices:                        # ...and the devices that draw on them in another
  - name: device-1
    consumesCounters:
    - counterSet: gpu-1-counters
      counters:
        memory:
          value: 6Gi
  - name: device-2
    consumesCounters:
    - counterSet: gpu-1-counters
      counters:
        memory:
          value: 6Gi

That is the honest representation of a MIG-capable GPU: overlapping logical devices that cannot all exist at once. CounterSets must be published in separate ResourceSlices from the devices that consume them, and a device can only draw from counter sets in its own pool. With the NVIDIA driver, MIG slices appear under the mig.nvidia.com DeviceClass with a profile attribute such as 1g.5gb and a parentUUID attribute, and the driver's guide shows a matchAttribute constraint on parentUUID to force two slices onto one physical GPU.

# Assembled by me from the NVIDIA MIG guide's description, not copied.
# Two MIG slices that must sit on the same physical GPU.
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: two-mig-slices
spec:
  spec:
    devices:
      requests:
      - name: small
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: device.attributes['gpu.nvidia.com'].profile == '1g.5gb'
      - name: medium
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: device.attributes['gpu.nvidia.com'].profile == '2g.10gb'
      constraints:
      - requests: ["small", "medium"]
        matchAttribute: "gpu.nvidia.com/parentUUID"

The second claim below is assembled from that guide's description and the DRA API shape, not copied from a page, so check it against the guide before use. Profile names depend on the GPU model.

Device taints: pulling a bad GPU out without draining the node

A device taint works like a node taint, with a key, a value and an effect. NoSchedule makes tainted devices invisible to allocation, NoExecute also evicts Pods already using them through the device taint eviction controller, and None is ignored by the scheduler and exists for signalling. Claims can tolerate taints, and an eviction can be delayed by tolerating for a duration. Admins create taints with a DeviceTaintRule that selects devices by DeviceClass, driver, pool or device name, and a rule with no selector taints nothing, on purpose, so a forgotten selector cannot evict everything.

# Dry run first: effect None changes nothing, but the controller reports
# how many devices it selects and how many Pods NoExecute would evict.
apiVersion: resource.k8s.io/v1
kind: DeviceTaintRule
metadata:
  name: example
spec:
  deviceSelector:
    driver: dra.example.com      # no selector at all would taint nothing, on purpose
  taint:
    key: dra.example.com/unhealthy
    value: Broken
    effect: None                 # read the status message, then change to NoExecute

# kubectl describe devicetaintrules
# kubectl wait --for=condition=EvictionInProgress=false DeviceTaintRule/example

The None effect gives you a dry run. The docs describe creating the rule with effect None, reading the message for how many devices are selected and how many Pods would be evicted, then editing it to NoExecute. One honest wrinkle: the 1.36 post lists device taints as beta, while the current docs mark taints and DeviceTaintRules as Stable since 1.37. Read the status for your own cluster version, not for this post's date.

What was stable in 1.36 and what the docs say now

Maturity is where DRA posts go stale fastest, so here is the 1.36 post set against the current concept pages I read. Where the two differ, the page is newer, and your cluster's version decides which one applies.

Extended resource support is the quiet one for adoption. A DeviceClass can carry an extendedResourceName, so a Pod that asks for example.com/gpu: 2 in the old style is satisfied by DRA devices on some nodes and by a device plugin on others. Operators can migrate the cluster without making every application team rewrite manifests first.

# From the DRA API page: a DeviceClass that answers old-style requests.
# A Pod asking for "example.com/gpu: 2" can now land on DRA-managed nodes.
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type == 'gpu'
  extendedResourceName: example.com/gpu

What bites: the named trade-offs

The first limit is in the docs plainly: the scheduler does not support preemption for DRA resources. A high-priority Pod that needs a DRA device cannot evict a running Pod that holds it, and stays Pending until the device is freed. A fallback list makes this less likely to hurt, because there are more ways to be satisfied, but it does not remove it.

The second is that the prioritized list is decided per Pod. The docs say that if a Pod belongs to a ReplicaSet or similar grouping, you cannot rely on every member having the same subrequest chosen, so the workload must accommodate it. Eight replicas can come up as a mix of one H100 and two A100s. An inference server with a fixed tensor-parallel size or a fixed memory budget needs to tolerate that or be split into separate claims.

Two smaller ones from the MIG guide: on Ampere cards such as the A100, MIG mode has to be enabled before the driver starts because toggling it needs a GPU reset, and if it is off the driver falls back to full-GPU allocation and advertises no partitions. A claim that asks for a MIG profile then simply finds nothing, which looks like a scheduling problem and is not.

What to do on Monday

Start by establishing what your cluster actually supports, then write the smallest fallback claim you could live with. The first commands are read-only.

1. Check the server version. The task pages require v1.34 or later, and the prioritized list is stable from 1.36.
2. Confirm DRA answers: kubectl get deviceclasses should return a list or No resources found, not a missing resource type error.
3. Dump your ResourceSlices and read the real attribute names and product strings before writing any CEL.
4. Write a two-entry firstAvailable claim whose second choice your workload really runs on, and apply it to a test namespace.
5. Before touching a production GPU, create a DeviceTaintRule with effect None and read its message for what a NoExecute would evict.

# 1. Server version: task pages need v1.34+, prioritized list is stable from 1.36
kubectl version

# 2. Is DRA answering? "No resources found" is fine; "doesn't have a resource type" is not
kubectl get deviceclasses

# 3. What do the drivers actually publish? Read real attribute names and product strings
kubectl get resourceslices
kubectl get resourceslice <name> -o yaml

# 4. Apply the fallback claim to a test namespace, then drill down if it hangs
kubectl apply -n dra-test -f h100-else-a100.yaml
kubectl describe resourceclaims -n dra-test

# 5. Rehearse an eviction before it is real (effect: None, then read the message)
kubectl apply -f taint-dry-run.yaml
kubectl describe devicetaintrules

Do not add DRA to a workload that does not need a choice. If the fleet is one card model and a count satisfies you, the device plugin still works, and the docs describe the migration path as gradual, with the maintainers' stated goal being to move users from the device plugin to DRA over time.

The rule I carry away: ask for a capability with an ordered fallback, make sure the workload behind every entry is actually valid, and check each feature's maturity against your own cluster version rather than against the date of the blog post you read it in.