Kubernetes Gang Scheduling Beta in 1.37: What Works, What Not

Picture a distributed training Job that needs eight Pods at once. The default scheduler places Pods one by one, so it can bind six, fail on the last two, and leave six expensive accelerators held by a job that cannot start. That is the deadlock gang scheduling exists to prevent, and until now the practical answer was to install a second system.

Kubernetes v1.37 brings an in-tree answer to Beta. I read the v1.37 workload-aware scheduling release post, the new gang scheduling documentation, the v1.36 post it builds on, and the Kueue and Volcano overview pages. I did not run a v1.37 cluster, so there are no benchmarks here, only what the sources state, with the gaps marked.

What reached Beta in 1.37, and what is still Alpha

The release post says the core Workload and PodGroup APIs, which enable gang scheduling, along with workload-aware preemption and shared DRA ResourceClaims for PodGroups, graduated to Beta. It also says both the Beta and the Alpha features are disabled by default and need manual enablement. The gang scheduling documentation agrees: Beta since v1.37, disabled by default. The split matters because the Beta path is one switch while each Alpha feature has its own.

Two details are easy to miss. The separate WorkloadAwarePreemption gate from 1.36 was merged into GenericWorkload, so one gate now carries both gang scheduling and workload-aware preemption. And enabling CompositePodGroup on the kube-controller-manager also requires TopologyAwareWorkloadScheduling there, so the hierarchy feature cannot be switched on alone.

The object model: Workload is the template, PodGroup is the runtime

Since 1.36 the Workload object is a static template and the PodGroup holds runtime state. The 1.36 post gives the reason: status updates can be sharded per replica, and the scheduler can read the PodGroup directly without watching the Workload. A controller writes the Workload, stamps out PodGroups from its templates, and sets spec.schedulingGroup.podGroupName on each Pod. The sketch below is a flat gang of four workers, assembled from the field names in the 1.37 post with the composite parent removed. Treat it as something to test, not a manifest copied from the release notes.

# kind-was.yaml: a throwaway cluster to try the Beta gate.
# Use a node image built for v1.37; the gate does not exist before it.
# I have not run this file. The gate and API group names are from the
# v1.37 release post; the kind keys are kind's standard v1alpha4 config.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  GenericWorkload: true
runtimeConfig:
  "scheduling.k8s.io/v1beta1": "true"
---
# The template: a Workload names the group and its gang policy.
apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: training-workload
  namespace: some-ns
spec:
  podGroupTemplates:
    - name: workers
      schedulingPolicy:
        gang:
          # Schedulable only if 4 Pods can be placed together.
          minCount: 4
---
# The runtime object. Create it BEFORE the Pods: the scheduler keeps the
# Pods in PreEnqueue until this exists and 4 Pods have been created.
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-workers-pg
  namespace: some-ns
spec:
  workloadRef:
    workloadName: training-workload
    templateName: workers
  schedulingPolicy:
    gang:
      minCount: 4
---
# Each Pod links to the PodGroup, not to the Workload.
apiVersion: v1
kind: Pod
metadata:
  name: worker-0
  namespace: some-ns
  labels:
    app: gang-test
spec:
  schedulingGroup:
    podGroupName: training-workers-pg
  containers:
    - name: worker
      image: busybox:1.36
      command: ["sleep", "3600"]
      resources:
        requests:
          cpu: "2"

Create the PodGroup before the Pods. The documentation says the scheduler holds Pods in the PreEnqueue phase until the referenced PodGroup exists and at least minCount Pods have been created, so a missing PodGroup keeps the whole gang out of the active queue rather than failing loudly.

How the scheduler holds and places a gang

Once the quorum is met, the scheduler runs a PodGroup scheduling cycle and makes one atomic decision. Its GangScheduling plugin implements a PlacementFeasible extension point, called for each evaluated Pod, that compares successfully placed Pods, including ones scheduled in earlier cycles, against minCount. If at least minCount placements are found the Pods are bound. If not, none of them are, and they wait in the unschedulable queue so other workloads can use the capacity in the meantime.

1.37 also changed what gets queued. Previously every member Pod was queued individually, even inside a PodGroup. Now only the top-level PodGroup object is queued, so all its Pods share the same queueing behaviour, and the post describes this as groundwork for more advanced PodGroup queueing strategies later.

The documentation carries a caveat worth reading twice. The scheduler never admits fewer Pods than minCount during initial placement, but the running count can later fall below it if Pods are deleted or evicted, or if minCount is raised. After that the scheduler places extra Pods only when scheduled plus newly feasible Pods reaches minCount. Gang scheduling is a guarantee about the first placement, not an invariant for the life of the workload.

A Job with an explicit gang policy

The part most teams will touch first is the Job API. The v1.37 Job has a .spec.scheduling field built from four pieces: schedulingPolicy, schedulingConstraints, disruptionMode and resourceClaims. This is the example from the release post, with a runnable container filled in where the post elides it.

# From the v1.37 release post. gang {} omits minCount, which then
# defaults to parallelism (8). Needs the Alpha WorkloadWithJob gate on
# kube-apiserver and kube-controller-manager.
apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training-job
spec:
  parallelism: 8
  completions: 8
  scheduling:
    schedulingPolicy:
      gang: {}
    schedulingConstraints:
      topology:
      - key: topology.kubernetes.io/zone
    disruptionMode:
      all: {}
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: busybox:1.36
        command: ["sleep", "3600"]

Omitting gang minCount defaults it to parallelism, 8 here. Leaving out .spec.scheduling, or schedulingPolicy inside it, selects the basic policy, which behaves like Job scheduling today. For each Job the controller compiles the configuration into a Workload and a PodGroup owned by the Job and sets the schedulingGroup on every Pod it creates. After creation .spec.scheduling is immutable, except that gang minCount can be updated, which lets you resize a running gang.

This integration is Alpha, behind WorkloadWithJob, and it covers the plain Job. For JobSet and LeaderWorkerSet the post points to CompositePodGroup, and lists deeper integration with hierarchical orchestrators such as JobSet as a planned focus, not something shipped.

Hierarchies and multi-level topology, in Alpha

CompositePodGroup extends the model from one flat group to a tree. A Workload gains spec.compositePodGroupTemplates, each carrying a gang policy with minGroupCount, and nests podGroupTemplates or further composite templates beneath it. The scheduler treats the whole tree as one scheduling unit. The release post's multi-level example pins the root to one availability zone and each child group to one rack inside that zone.

# From the v1.37 release post: a two-level hierarchy, zone for the whole
# workload, rack for each child group. Alpha: needs the CompositePodGroup and
# TopologyAwareWorkloadScheduling gates, and scheduling.k8s.io/v1alpha3.
apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: multi-level-tas-workload
  namespace: job-ns
spec:
  compositePodGroupTemplates:
  - name: root
    schedulingPolicy:
      gang:
        minGroupCount: 2          # both child groups, or nothing
    schedulingConstraints:
      topology:
      - key: topology.kubernetes.io/zone
    podGroupTemplates:
    - name: workers
      schedulingPolicy:
        gang:
          minCount: 8
      schedulingConstraints:
        topology:
        - key: topology.example.com/rack
    - name: driver
      schedulingPolicy:
        gang:
          minCount: 1
      schedulingConstraints:
        topology:
        - key: topology.example.com/rack

Evaluation is recursive and all-or-nothing: if the root cannot satisfy its policy, no Pod in the hierarchy is bound. Topology constraints resolve top-down, so a child group's candidate domains sit inside the placement assumed for its parent. Preemption extends too: disruptionMode single lets child groups be preempted independently, while all evicts every Pod across the hierarchy together.

One wrinkle in the post's own examples: the runtime CompositePodGroup is scheduling.k8s.io/v1alpha3 while the leaf PodGroups are v1beta1, and the root links its children through parentCompositePodGroupName. The post's roadmap puts topology-aware scheduling and CompositePodGroup at Beta in v1.38, which is a plan, not a commitment.

Where Kueue and Volcano still belong

Native gang scheduling answers one question: can this group of already created Pods all be placed? Kueue answers an earlier one. Its overview says it decides when a job should wait, when it should be admitted, meaning Pods can be created, and when it should be preempted. My reading is that the two sit at different moments: Kueue gates Pod creation against quota, and gang scheduling gates binding after the Pods exist.

The Kubernetes post is explicit about direction. Under what is next it lists making Kueue fully aware of workload-aware scheduling features in the near term, and in the long term Kueue using the native APIs as its engine for gang scheduling and topology-aware placement. That reads as convergence, but it is a stated intention for v1.38 and beyond, not a shipped feature, and I found no compatibility statement for today's Kueue.

My decision rule from the sources: if you need quotas between teams, fair sharing or admission control, you still need Kueue or Volcano, and native gang scheduling does not replace them. If you run one team's training Jobs on a cluster you control and only the deadlock hurts, the native Beta path is worth a test cluster now.

Where it breaks: the trade-offs I would write down first

A Beta feature that is off by default has costs the headline does not mention. These are the ones the sources support.

• Three control plane components must agree. The Beta gate goes on kube-apiserver, kube-controller-manager and kube-scheduler, and manifests must use scheduling.k8s.io/v1beta1. I did not research whether managed Kubernetes services expose control plane feature gates.
• Placement is not guaranteed for awkward groups. The 1.36 post says that for heterogeneous groups, and for groups with inter-Pod dependencies such as affinity, finding a valid placement is not guaranteed even when one exists. I found nothing in the 1.37 post that lifts this.
• Gang scheduling is not admission control. A gang that cannot fit moves to the unschedulable queue and waits. Nothing in these APIs rations capacity between teams.
• Alpha versions break. v1alpha2 was entirely replaced by v1alpha3 with breaking changes around disruptionMode, whose values were renamed: PodGroup became all and Pod became single.
• A disabled DRA gate now creates nothing. With DRAWorkloadResourceClaims off, a Pod whose claim matches its PodGroup's claim gets no ResourceClaim at all in 1.37, where 1.36 created one per Pod. That avoids a flood of claims but I would expect a Pod that depends on a claim to stay Pending.

The rename in practice, using the field forms shown in the 1.36 and 1.37 posts:

# Wrong: a 1.36 alpha manifest applied to a 1.37 cluster.
# v1alpha2 was replaced, and the old disruptionMode value was renamed.
apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
spec:
  disruptionMode: PodGroup

# Right: the 1.37 beta shape. all replaces PodGroup, single replaces Pod.
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
spec:
  disruptionMode:
    all: {}

What to do on Monday

A short list that costs one throwaway cluster and an afternoon:

1. Sort your jobs into two piles: those that are useless unless every Pod runs, such as distributed training, and those that tolerate partial progress. Only the first pile is a candidate.
2. Build a throwaway cluster on a v1.37 node image with GenericWorkload and the v1beta1 API group on, and ask for more capacity than it has.
3. Confirm the behaviour the docs promise: every Pod stays Pending with no node assigned, and the PodGroup, not the Pod, carries the status.
4. Check the gate state on the control plane you actually run before planning anything, and note which of your clusters cannot turn it on.
5. Leave Kueue and Volcano in place. Re-read the Kubernetes v1.38 workload-aware scheduling post when it lands, because GA for Workload and PodGroup and Kueue alignment are both on its stated roadmap.

Resizing a running gang is a one-line patch on the Job, since gang minCount is the one mutable scheduling field:

# 1. Is the beta API group served at all?
kubectl get --raw /apis/scheduling.k8s.io/v1beta1 | head -c 300
kubectl api-resources --api-group=scheduling.k8s.io

# 2. Gate state: the API server exports each gate as a metric.
kubectl get --raw /metrics | grep 'kubernetes_feature_enabled{name="GenericWorkload"'

# 3. The experiment: request more CPU than the cluster has, then look.
kubectl apply -f gang-test.yaml
kubectl get pods -l app=gang-test -o wide   # expect Pending, NODE <none>
kubectl get podgroups -n some-ns
kubectl describe podgroup training-workers-pg -n some-ns

# 4. Resize a running Job gang. minCount is the one mutable scheduling field.
kubectl patch job distributed-training-job --type merge \
  -p '{"spec":{"scheduling":{"schedulingPolicy":{"gang":{"minCount":6}}}}}'

Native gang scheduling in v1.37 is real but narrow: an off-by-default Beta that guarantees all-or-nothing placement for a group of Pods, not a replacement for the systems that decide who gets capacity and when. Test it where a deadlock actually hurts, rewrite Alpha manifests before upgrading, and keep your queueing layer until the Kubernetes project says otherwise.