Kubernetes HPA Scale to Zero for GPU Queue Workers (1.37)

The queue worker every team eventually builds spends most of its life waiting. Jobs arrive in bursts, the Deployment sits at one replica around the clock, and if each replica reserves a GPU, the idle hours are the expensive ones. Until this release the built-in HorizontalPodAutoscaler could not fix that, because its floor was one replica.

Kubernetes v1.37 moves the floor. This post is a close reading of what the release post, the HPA documentation, the feature's own blog post and KEP-2021 say about scaling to zero: the config that works, the condition that keeps it honest, the delays you pay, and where it stops. I have not run 1.37 in production, so every number below is quoted from those sources, and where I reason past them I say so.

What 1.37 changed, and what it did not

The v1.37 release notes say HPA scale to zero was first introduced in Kubernetes v1.16 and graduates to Beta in v1.37, where it is now enabled by default. The gate is HPAScaleToZero, and the HPA documentation says it must be on in both kube-apiserver, which allows minReplicas 0, and kube-controller-manager, which does the scaling. The default is the real change: the feature's own blog post says that before 1.37, scaling from zero required an add-on, an external component or the Alpha gate.

The less obvious change landed a release earlier. The blog post's timeline says v1.36 added the ScaledToZero condition and the controller behaviour that separates an automatic scale-down from a manual pause. That condition, not the replica count, is what makes the feature safe, and section four covers why. Beta is also not GA: the post says the next step is gathering operational feedback before considering it, and the KEP tracking issue lists no stable target version.

Why only object and external metrics qualify

Scaling to zero on CPU or memory is not supported, because those metrics depend on active Pods. The HPA documentation puts it the same way: resource metrics can only be measured on running Pods, so at zero replicas there is nothing to measure and nothing that could ever ask for the first replica back. Setting minReplicas to 0 requires at least one object or external metric, and the API server rejects the HPA otherwise.

Object and external metrics do not have that problem. The docs describe them as a single metric fetched for the object in question and compared with a target. A queue length lives in the broker, not in the workers, so it keeps reporting after the workers are gone. That is the whole design: the signal that wakes a workload must outlive the workload. KEP-2021 states the goal narrowly, scaling to and from zero for object and external metrics, and lists resource metrics and request buffering at the Service level as non-goals.

A config that works: queue lag, Prometheus Adapter, one HPA

The Kubernetes blog's worked example assumes Prometheus already collects a series called queue_consumer_lag with namespace and name labels, and exposes it through the Prometheus Adapter's externalRules. The adapter's own docs explain the detail that trips people up: HPAs are namespaced, so the adapter adds a namespace label to the query, and the External Metrics API path carries the namespace. Check the metric through the API server before you write the HPA.

# 1. prometheus-adapter values: expose queue_consumer_lag as an EXTERNAL metric.
#    Shape taken from the Kubernetes v1.37 HPA post and the adapter's
#    external-metrics doc (v0.12.0).
externalRules:
- seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
  metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
  resources:
    overrides:
      namespace:
        resource: namespace   # HPAs are namespaced, so the adapter scopes by namespace

# 2. Prove the metric through the API server before any HPA exists.
#    If this does not print the current value, an HPA cannot wake from zero.
kubectl get --raw \
  '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

The HPA below follows the blog's example: a Deployment named queue-worker, zero to ten replicas, one replica requested per 30 queued tasks. The Deployment, the GPU limit and the grace period are mine, added to show the case the feature is sold on. Replace the GPU line with whatever extended resource your device plugin advertises.

# Deployment: leave spec.replicas OUT. It defaults to 1, which gives the HPA
# a workload to scale down, and a later kubectl apply will not reset the count.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: queue-worker
spec:
  selector:
    matchLabels:
      app: queue-worker
  template:
    metadata:
      labels:
        app: queue-worker
    spec:
      terminationGracePeriodSeconds: 600   # my choice: long enough to finish a task
      containers:
      - name: worker
        image: registry.example.com/queue-worker:1.0.0
        resources:
          limits:
            nvidia.com/gpu: 1   # swap for whatever extended resource your device plugin advertises
---
# HPA: the shape of the Kubernetes v1.37 blog example. One replica per 30
# queued tasks, from 0 up to 10.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0          # needs at least one object or external metric below
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"

Three details in the blog post matter more than the YAML. Start the Deployment with at least one replica, because the HPA will not wake a workload it did not scale down itself. The downscale stabilisation window defaults to five minutes. And the API server rejects minReplicas 0 on an HPA that uses only resource metrics. The docs also advise removing spec.replicas from the manifest once an HPA owns the Deployment, since a later kubectl apply would otherwise reset the count to that value and cause thrashing.

Zero versus paused: the ScaledToZero condition

Before this feature, replicas 0 meant one thing: a human had paused autoscaling. The KEP says that worked because the HPA could never reach zero by itself. Once minReplicas 0 is legal, a Deployment at zero is ambiguous, so the HPA records a ScaledToZero condition. It is True when the HPA moves a workload from one replica to zero, and False with the reason NotScaledToZero after it scales back up. A workload at zero without ScaledToZero True stays paused.

# Who put this workload at zero: the HPA, or a human?
kubectl get hpa queue-worker \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

# Reading it, per the HPA docs and the v1.37 blog post:
#   ScaledToZero=True                   the HPA scaled it down and will wake it
#   ScaledToZero=False NotScaledToZero  it was scaled back up
#   no ScaledToZero=True, replicas 0    a manual pause: the HPA will NOT wake it
#   ScalingActive=False FailedGetExternalMetric
#                                       the metric is gone; nothing can scale up

kubectl describe hpa queue-worker      # same conditions, plus recent events

The KEP is candid about the sharp edge this leaves. Applying a new Deployment with replicas 0 next to an HPA with minReplicas 0 never scales, because the HPA did not put it there. The KEP says this is detectable through the existing ScalingActive condition and that an explicit pause feature could replace the implicit one later. I would make that the first check whenever a worker is stuck at zero with work waiting.

The cold-start budget: where the delay comes from

The release blog names cold-start time as the trade-off: the HPA has to observe the metric, schedule a Pod and start the application. The Kubernetes sources put numbers on the HPA's own part only, and I will not invent the rest. The table lists what they do state.

Two readings follow. First, KEP-2021's own SLO note says the first replica appears on the next reconcile after the metric crosses the threshold, plus whatever freshness the metrics pipeline adds, with no extra scale-up delay. Second, everything after that is outside the HPA: image pull, GPU node availability, model load. The docs point to node autoscaling for the right number of nodes; my inference is that a GPU node that has also scaled away adds its own wait, which these sources do not quantify. Measure the whole path on your own cluster before you promise anyone a latency.

# Added to the HPA spec above. The default scaleDown window is 300 s and the
# docs describe it as a rolling maximum of past desired states. 900 is MY
# number for bursty queues, not a recommendation from the Kubernetes docs:
# it trades a longer idle tail for fewer cold starts between bursts.
behavior:
  scaleDown:
    stabilizationWindowSeconds: 900
  scaleUp:
    stabilizationWindowSeconds: 0   # the documented default: scale up at once

Native HPA or KEDA: what you give up and what you stop installing

KEDA already offers scale to zero, so the honest question is what the native feature changes. The KEDA docs describe two phases. In the activation phase the KEDA operator decides whether a workload goes from or to zero, and that only applies to 0 to 1. In the scaling phase the HPA takes over for 1 to N from metrics KEDA exposes. The native feature puts the 0 to 1 decision inside the HPA controller itself, but only for a metric you already serve.

My reading: if you already run a metrics adapter and have one well-understood queue, the native feature removes a component. If you need a wake-up threshold different from the scaling target, or an idle floor, the KEDA docs show both and the HPA docs I read show neither. An assignee on the KEP issue answered a question about separate idle and active floors by saying it would be better tracked as a new KEP.

Where it stops working

Five failure modes, each taken from the sources rather than from an outage of mine:

• The metric disappears. The HPA reports ScalingActive False with a reason such as FailedGetExternalMetric, and a workload at zero stays there until you restore the metric or scale it by hand.
• A person or a pipeline sets replicas to 0. Without ScaledToZero True the HPA reads that as a pause and will not wake it, and a freshly applied Deployment at zero next to minReplicas 0 never scales.
• The control plane is half upgraded. A controller manager with the gate off treats replicas 0 as a manual pause, so the blog says to wait until both components support the feature before creating HPAs with minReplicas 0.
• The workload takes HTTP traffic. Services do not buffer requests, and request buffering is an explicit KEP non-goal, so a request-driven workload needs a separate buffering layer.
• You roll back. Set affected HPAs to minReplicas 1 or higher and scale any workload at zero to at least one replica before disabling the gate or downgrading, or the workload can stay stuck at zero.

There is one more failure the sources do not name, and it follows from how the documented pieces combine. The metric in the example is queue lag, the tasks still waiting. A task a worker has already taken is no longer waiting. The docs say the stabilisation window uses the highest desired value over the past window, five minutes by default, so a long job can outlive that window with the queue empty and the HPA free to remove the Pod running it. This is my reasoning from documented behaviour, not something I measured.

What to do on Monday

If you are on 1.37 the feature is already on, so start with a read-only audit and only then pick one workload. Five steps, in order:

1. Find every HPA that already sets minReplicas 0 and every Deployment sitting at zero replicas. The first can now scale by itself; the second may be paused or may be waiting.
2. Pick one queue consumer with long idle stretches and expensive Pods, and prove its metric through the External Metrics API before you touch an HPA.
3. Apply the HPA with minReplicas 0 in staging, drain the queue, watch ScaledToZero turn True, then enqueue one task and time the gap from the message to the first Ready replica.
4. Check skew and rollback. On a mixed control plane, wait until both components support the feature. Before any downgrade, set minReplicas to 1 and scale zero-replica workloads up.
5. Add the alert below: the queue has work, the HPA cannot act. It catches a broken adapter before a user notices a stalled job.

# Step 1: every HPA that may already scale to zero, and every Deployment at zero.
kubectl get hpa -A -o json \
  | jq -r '.items[] | select(.spec.minReplicas == 0) | .metadata.namespace + "/" + .metadata.name'
kubectl get deploy -A -o json \
  | jq -r '.items[] | select(.spec.replicas == 0) | .metadata.namespace + "/" + .metadata.name'

# Step 5: page when the queue has work but the HPA cannot wake the workers.
# Metric names are the ones KEP-2021 points at (kube-state-metrics); the label
# names are kube-state-metrics' own, so confirm them on your version.
groups:
- name: scale-to-zero
  rules:
  - alert: QueueHasWorkButWorkersCannotWake
    expr: |
      max(queue_consumer_lag{name="worker_tasks"}) > 0
      and on()
      max(kube_horizontalpodautoscaler_status_condition{
            horizontalpodautoscaler="queue-worker",
            condition="ScalingActive", status="false"}) == 1
    for: 5m

Choose the first workload by two tests: an idle hour costs real money, and a cold start does not hurt anyone. The release blog frames the fit as work that can wait in a durable queue, and that is the filter I would apply.

The rule I would carry out of this: scale to zero is a promise about the metric, not about the workload. The HPA can idle a queue consumer only because the queue's length outlives the workers, so own that metric's path, its freshness and its alerting first, and check whether any job outlives the downscale window second. Get those right and 1.37 removes an add-on from your cluster. Get them wrong and the failure looks exactly like an empty queue.