Volcano Scheduler GPU Gang Preemption: v1.15 DRA Queue Quota

Picture a GPU cluster where three distributed training jobs are running and a higher-priority gang arrives. The scheduler needs room, so it evicts one Pod from each running job. Capacity is freed, yet all three jobs now stall on a missing worker, and the new gang still cannot gather enough Pods to start. That exact cascade is what the Volcano v1.15 release post sets out to remove, and it is why I read the release end to end.

I have not run v1.15 on a GPU cluster, so this post makes no benchmark claim. It is a close reading of the Volcano v1.15.0 release post, the GitHub release note and the scheduling-gates user guide, set against the Kubernetes documentation for native gang scheduling and workload-aware preemption. Where a sentence says alpha, the source says alpha.

Why per-Pod preemption breaks distributed jobs

A gang job is only useful while its minimum number of Pods run together. The release post is specific about the legacy behaviour: per-Pod preemption can evict one Pod from each of several running training jobs, which frees capacity on paper but breaks every one of those jobs, while the preemptor still lacks enough Pods to reach its own minAvailable.

Volcano's gang plugin already gave all-or-nothing placement for a job that has not started: its docs describe running the scheduling action for a job's Pods only when the minimum number can run. According to the release post, the missing piece was the eviction side, which did not treat the victim's gang as a unit. Everything in v1.15 that matters to a GPU cluster is a variation on one theme: stop deciding about single Pods when the thing that matters is a gang, a queue or a device class.

Gang-aware preemption: victims by job, preemptor as a whole

The release splits the change into two sides. On the victim side, candidates are organised per job or gang, and each job's Pods are classified as surplus replicas (above minAvailable) or critical replicas. Surplus replicas are preferred, because evicting them does not break the victim job. On the preemptor side, the scheduler accumulates reclaimable resources step by step; once the total covers the whole preemptor gang, it runs a placement simulation on the projected resource view and commits evictions only if the whole gang fits.

# Wrong: an illustration of what the upgrade notes advise against.
# Legacy per-Pod eviction and gang-aware eviction in one action list.
actions: "enqueue, allocate, backfill, preempt, reclaim, gangPreempt, gangReclaim"

# Right: the example from the v1.15 release post. gangPreempt and gangReclaim
# REPLACE preempt and reclaim; they do not sit beside them.
actions: "enqueue, allocate, backfill, gangPreempt, gangReclaim"
tiers:
  - plugins:
      - name: priority     # orders gangPreempt victims (per the GitHub release note)
      - name: gang
      - name: drf
      - name: predicates
      - name: nodeorder
      - name: binpack

The GitHub release note adds details the blog post omits. The two actions order victims differently: gangPreempt by priority, gangReclaim by queue fairness. An allowWholeBundle option, default true, controls whether whole-job victim bundles can be chosen. With HyperNode topology enabled, victim search is limited to the selected topology scope, so preemption does not cross topology domains. The release post also says the team will evaluate merging this mechanism with the legacy preempt and reclaim actions.

Two cautions sit in the upgrade notes. The feature is Alpha, and configuring gangPreempt and gangReclaim together with legacy preempt and reclaim in one action list is not recommended. Choose one set per scheduler, and read the config block above as a replacement, not an addition.

DRA queue quota: one capacity model for CPU, memory and devices

Earlier Volcano versions could schedule Pods that use DRA resources, but queue quota did not cover DRA ResourceClaims. In v1.15 the capacity plugin counts them against the queue's existing capability, deserved and guarantee fields, so there is no second quota API to maintain. Two kinds of control exist: a whole-device count per DeviceClass, and a consumable dimension such as virtual GPU cores or memory.

# Scheduler config: the capacity plugin must own DRA accounting.
tiers:
  - plugins:
      - name: capacity
        arguments:
          capacity.DynamicResourceAllocationEnable: true
          capacity.DRAConsumableCapacityEnable: true   # vGPU cores / memory dimensions
---
# Queue from the release post. The keys are the part to learn:
#   deviceclass/<DeviceClass>        -> whole-device count
#   <dimension>.deviceclass/<Class>  -> consumable capacity (cores, memory)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: ml-team
spec:
  capability:
    cpu: "100"
    memory: "200Gi"
    "deviceclass/gpu.nvidia.com": "8"
    "cores.deviceclass/hami-core-gpu.project-hami.io": "800"
    "memory.deviceclass/hami-core-gpu.project-hami.io": "320Gi"

The key shape is the part to memorise. The GitHub release note gives it as deviceclass followed by a slash and the DeviceClass name for whole-device counts, and the dimension, a dot, then the same deviceclass key for consumable capacity. In the post's example the ml-team queue allows 8 whole devices of gpu.nvidia.com and, for a HAMi vGPU class, 800 cores and 320Gi of memory. Those numbers illustrate the syntax; they are not a sizing recommendation.

When several Pods reference one shared ResourceClaim, Volcano deduplicates it so the device is not counted once per Pod; the post calls this case out explicitly. The prerequisites are Kubernetes DRA support and an available DRA driver, and the GitHub note says some DRA allocation modes sit outside the first quota-accounting scope, so test yours. Separately, DRA scheduling integration is now on by default to match Kubernetes 1.34 and later, and predicate.DynamicResourceAllocationEnable set to false turns it off.

Scheduling gates: keep quota-blocked Pods away from the autoscaler

Cluster Autoscaler or Karpenter can read a Pod that is waiting for queue capacity as a shortage of nodes and scale up, which buys hardware the queue will not let you use. v1.15 adds Scheduling Gates for Queue Admission, Alpha. For Pods that opt in, the Volcano webhook injects a scheduling gate at creation, the scheduler removes it once the queue capacity check passes, and while the gate is present the Pod is invisible to autoscalers.

# 1. Enable the gate on BOTH the scheduler and the webhook-manager.
helm install volcano volcano/volcano --namespace volcano-system --create-namespace \
  --set custom.scheduler_feature_gates="SchedulingGatesQueueAdmission=true" \
  --set custom.admission_feature_gates="SchedulingGatesQueueAdmission=true"

# 2. Opt a Pod in. Without the annotation nothing changes for it.
apiVersion: v1
kind: Pod
metadata:
  name: queue-gated-pod
  annotations:
    scheduling.volcano.sh/queue-allocation-gate: "true"
spec:
  schedulerName: volcano
  containers:
    - name: worker
      image: nginx

# 3. While the queue is full this prints the gate; once admitted it prints nothing.
kubectl get pod queue-gated-pod -o jsonpath='{.spec.schedulingGates}'

This reuses a stable Kubernetes primitive. The Kubernetes page on Pod scheduling readiness lists it as stable since v1.30 and says Pods stuck in a missing-resources state churn the scheduler and integrators such as Cluster Autoscaler. Gates can be set only at creation and removed in any order later. The Volcano user guide adds that if other controllers have added their own gates, Volcano waits until only its gate remains before removing it.

After ungating, a Pod that fits no node is marked Unschedulable, which correctly triggers the autoscaler. The gate therefore separates two questions that used to look identical from outside: is the queue willing to admit this Pod, and does the cluster have a node for it.

Volcano 1.15 against native Kubernetes gang scheduling

Kubernetes now ships its own answer to the same problem: PodGroup, gang scheduling and workload-aware preemption, with gang scheduling Beta since v1.37. The table sets the two side by side using each project's own documentation. A thin cell for one side is a statement about the pages I read, not a claim that the capability cannot exist elsewhere.

The pattern is that native Kubernetes gives you the primitive inside kube-scheduler, while Volcano gives you the queue around it. Native is ahead on being in-tree and Beta; Volcano is ahead wherever the question is quota, fairness between teams or autoscaler behaviour. Neither source benchmarks the other, so I will not rank them on speed.

Trade-offs and where this will bite

Four limits come straight from the sources. Each is the kind that shows up in the first busy week rather than in a demo.

• Alpha means interfaces can move. Gang preemption, scheduling gates and the multi-sharding policy are all Alpha in 1.15, and the post says the team is still deciding whether to fold gang preemption into the legacy actions.
• Reserved queue capacity has no timeout. The user guide says Pods that are ungated but stay unschedulable keep their reserved queue capacity, which can block other Pods.
• The gate works only with the capacity plugin. Users of proportion may still see false scale-ups until Volcano issue 5271 is resolved.
• Upgrading changes defaults. DRA integration turns on by default, and the GitHub note says HyperNode gradient callbacks now take a SearchPurpose argument, so custom topology plugins need updating.

Native Kubernetes has its own sharp edges, from its lifecycle page. All Pods in a PodGroup must share the same schedulerName, priority and preemptionPolicy, or the scheduler rejects the group, and PodGroup status is not updated after initial placement. Also note the name collision: both projects now have an object called PodGroup, so spell out the API group, as in podgroups.scheduling.k8s.io, when debugging.

How I would choose between them

If you already run Volcano with queues, take 1.15 into staging first. DRA quota and queue-admission gates address problems you probably already have, and neither requires changing how jobs are written. Leave gang-aware preemption for a dedicated test scheduler until it leaves Alpha.

If you have one team, one cluster and no queue hierarchy, native gang scheduling and workload-aware preemption may be enough. They need the GenericWorkload gate and the v1beta1 API group, and they keep everything inside kube-scheduler, which is one fewer component to run and upgrade.

Do not run both on the same Pods. A PodGroup requires every Pod to use one schedulerName, so a gang belongs to one scheduler or the other. This is my reading of the documentation, not a tested migration path.

What to do on Monday

Start with the cheap, reversible steps and leave the Alpha switch for last.

1. Run the Volcano benchmark framework on Kind plus KWOK to record a baseline before changing anything.
2. Upgrade a staging cluster to 1.15.0 and check whether the DRA default changes how existing workloads place; set the predicate flag to false if it does.
3. Add one DRA key to a test queue, enable the two capacity arguments, and watch queue usage with a shared ResourceClaim.
4. Enable the queue-admission gate on both components and annotate one namespace's Pods, then watch for gated Pods that go Unschedulable.
5. On a separate scheduler, switch the action list to gangPreempt and gangReclaim and reproduce the cascade: two running gangs with surplus replicas and one higher-priority gang arriving.

# 1. Baseline first, on Kind + KWOK (run from the repo's benchmark directory).
cd benchmark
make setup VOLCANO_VERSION=v1.15.0
make test-gang-env JOBS=10 REPLICAS=100 MIN_AVAILABLE=100
make cleanup-all

# 2. Upgrade staging. DRA integration is ON by default from 1.15; to keep the
#    old behaviour set predicate.DynamicResourceAllocationEnable: false.
helm repo update
helm upgrade volcano volcano-sh/volcano --version 1.15.0

# 3. Look at what the queue thinks it holds before and after adding a DRA key.
kubectl get queues.scheduling.volcano.sh ml-team -o yaml

# 4. Two objects are both called PodGroup. Spell out the API group when debugging.
kubectl get podgroups.scheduling.k8s.io -A   # Kubernetes native (v1beta1 in 1.37)

Step 5 comes last because it is the only Alpha change that alters who gets evicted. If the reproduction shows the surplus replicas going first and no running gang dropping below minAvailable, you have evidence from your own workloads rather than from a release post.

The rule I take from v1.15: let the scheduler reason about the unit that actually fails. A gang fails as a gang, a quota is spent in devices and dimensions, and an autoscaler should only hear about shortages of nodes. Adopt the parts that are reversible now, and treat Alpha eviction logic as something you prove on your own workloads.