Autoscale vLLM on Kubernetes with KEDA: Queue Depth, Not GPU %

The first thing I look for in an inference autoscaler is what it is told to watch. A GPU graph that sits at a comfortable-looking number tells you nothing about the person whose request has been waiting for ten seconds, and a replica that takes minutes to become useful cannot be added after the complaint arrives.

This post is a close reading of the AWS EKS guide on autoscaling AI inference with HPA and KEDA, its companion page on finding thresholds, the KEDA and Kubernetes reference pages it depends on, and the llm-d v0.9.0 release notes. I have not run this on a cluster. Every number below was read from a page and is attributed to it, and where I am reasoning rather than quoting I say so.

Why queue depth and p95 latency, not GPU utilisation

AWS scales the vllm-inference-app Deployment on two Prometheus signals. Queue depth is described as the primary demand signal and p95 end-to-end latency as an SLO guardrail. GPU utilisation does not appear in the manifest at all. The vLLM metrics design page frames the underlying problem: it says a common use of the metrics is automated scaling, that the goal is to expose what is needed to detect the saturation point, and that this is a non-trivial topic.

My reasoning, not a claim from either page: a GPU reports how busy it is, not how many requests are waiting. In AWS's own load test the waiting count stayed near zero until the replica saturated and then jumped to the hundreds, which is a far cleaner step than anything a utilisation graph is promised to give. The table lists what each candidate signal means according to the pages I read.

Where 25 and 5 come from: the load test

The thresholds page uses Ministral-3-8B-Instruct-2512 on a g6e.4xlarge and k6 in a constant-arrival-rate model at 10, 20, 30, 40, 50, 60 and 70 requests per second, 60 seconds per rate with a 30-second pause between. It warms the GPU first with 100 sequential requests, because vLLM captures CUDA graphs, compiles and autotunes kernels and allocates KV cache pools on the first inferences while the GPU raises its clocks. A Ready replica is therefore not a warm replica.

The result, in the page's words: queue depth stayed near zero through 50 concurrent requests, then rose sharply at 60, peaking between roughly 150 and 350 queued requests, while p95 end-to-end latency went from a steady 2.5 seconds or so to 6 to 10 seconds. The page labels the steps in requests per second but describes them as concurrent requests, so I treat the 50-to-60 step as the knee and would not read more precision into it.

From the knee it derives three starting points: scale up when queue depth exceeds 25 for 30 to 60 seconds, scale up as a secondary signal when p95 exceeds 5 seconds, and scale down only when the queue is 0 and p95 is under 3 seconds for 5 minutes. The page says outright that these are a starting point and to repeat the exercise for your own model, GPU and request patterns. Here is the load generator that produces the curve.

// k6 script from the AWS thresholds page (stored in the vllm-loadtest-script ConfigMap).
// constant-arrival-rate holds the request RATE fixed, so when the GPU cannot keep up
// the excess shows up as vllm:num_requests_waiting instead of slowing the client down.
import http from 'k6/http';

// Random leading word varies the prompt so prefix caching does not flatter the result.
const words = ['revolution', 'technology', 'javascript', 'kubernetes', 'basketball'];

export const options = {
  noConnectionReuse: true,
  scenarios: {
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: __ENV.TARGET_RPS || 10,   // AWS steps this 10, 20, ... 70
      timeUnit: '1s',
      duration: __ENV.DURATION || '60s',
      preAllocatedVUs: 50,
      maxVUs: 500,
    },
  },
};

export default function () {
  const word = words[Math.floor(Math.random() * words.length)];
  const payload = JSON.stringify({
    model: __ENV.MODEL,
    prompt: 'Starting with the word "' + word + '", write a brief technical explanation about cloud computing concepts.',
    max_tokens: 100,
    temperature: 0.7,
  });
  http.post('http://vllm-inference-svc:8000/v1/completions', payload,
    { headers: { 'Content-Type': 'application/json' }, timeout: '240s' });
}

The ScaledObject, line by line

KEDA installs with Helm into its own namespace, and the ScaledObject below is AWS's manifest with my comments added. KEDA creates and manages a backing HPA with the same name, so you never write that HPA yourself. The comments mark the choices that matter.

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace
kubectl get pods -n keda   # operator, metrics-apiserver, admission-webhooks

kubectl apply -f - <<'EOF'
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-inference-app
  minReplicaCount: 1          # 1 means the scaler is always "active" (see activation section)
  maxReplicaCount: 5
  advanced:
    horizontalPodAutoscalerConfig:
      name: vllm-inference-app   # the HPA KEDA creates and owns; do not edit it by hand
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
          policies:
            - type: Pods
              value: 2
              periodSeconds: 60
          selectPolicy: Max
        scaleDown:
          stabilizationWindowSeconds: 300   # GPU pods are slow to start; do not shed fast
          policies:
            - type: Pods
              value: 1
              periodSeconds: 120
          selectPolicy: Min
  triggers:
    # Primary: demand exceeds capacity. AverageValue divides by replicas => 25 PER POD.
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
        # Wrong: sum(vllm:num_requests_waiting)  -> empty result when nothing is waiting
        # Right: add "or vector(0)" so the trigger always reports a number
        query: sum(vllm:num_requests_waiting) or vector(0)
        threshold: "25"
        activationThreshold: "1"
    # Guardrail: p95 end-to-end latency in seconds over a 1-minute rate window.
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
        query: histogram_quantile(0.95, sum(rate(vllm:e2e_request_latency_seconds_bucket[1m])) by (le)) or vector(0)
        threshold: "5"
EOF

kubectl get scaledobject vllm-inference-app   # READY True, ACTIVE False while idle
kubectl get hpa vllm-inference-app            # TARGETS shows both: x/25 (avg), y/5 (avg)

Three details carry the design. metricType AverageValue makes KEDA divide each metric across replicas, so 25 means 25 waiting requests per pod even though the query sums across all of them. The or vector(0) clause returns 0 instead of an empty result when nothing is waiting, which AWS says keeps the trigger from going inactive. And KEDA scales to whichever trigger asks for the most replicas, so either signal alone can add capacity.

Karpenter is the other half of the story on this cluster: AWS says it provisions a new GPU node if none is available, and the new replica becomes Ready only once the model has loaded into GPU memory. KEDA decides how many pods are wanted; it does not make them appear quickly.

Activation threshold versus scaling threshold

KEDA's scaling-deployments page describes two phases. The activation phase decides between 0 and 1 replicas from whether the scaler is active, and the scaling phase hands 1 to N to the HPA using the target value. Activation needs a value strictly greater than the activation threshold, it defaults to 0, and it takes priority: with threshold 10 and activationThreshold 50, a metric of 40 leaves the scaler inactive and scales to zero even though the HPA would otherwise want 4 instances.

Here is a catch I found by putting two pages side by side. The same KEDA page says that if minReplicaCount is 1 or more, the scaler is always active and the activation value is ignored. AWS's manifest sets minReplicaCount to 1, yet its text says activationThreshold 1 keeps the deployment at minReplicaCount until a request is waiting. Read together, the activationThreshold in that manifest documents intent but gates nothing. I have not tested which reading holds on a live cluster. Also note strictly greater than: an activation of 1 needs at least 2 waiting requests.

My inference: vllm:num_requests_waiting is emitted by vLLM pods, so with zero pods there is no series to cross any activation threshold, and a request sent to a Service with no endpoints never reaches a vLLM queue to be counted. Scale from zero needs a signal from something that sits in front of the pods, such as a gateway or a message queue, and KEDA's force-activation annotation exists for forcing a scale up by hand. For latency-sensitive GPU serving I would keep minReplicaCount at 1 and accept the idle cost, which is also what AWS's manifest does.

Fast up, slow down: the behaviour windows

The HPA takes the largest desired replica count across metrics, which is why a latency breach can add pods while the queue still looks fine. Kubernetes documents scale-down stabilisation as using the highest recommendation from the window, defaulting to 5 minutes, so AWS's 300 seconds matches that default. The scale-up side is tuned separately: a 30-second window and a policy of up to 2 pods per 60 seconds.

A little arithmetic of my own, not from either page: going from 1 to the maxReplicaCount of 5 needs four more pods, and at 2 per 60 seconds that is at least two policy periods before any node provisioning or model load. Coming back down, four removals at one per 120 seconds spread over at least about six minutes. The asymmetry is deliberate, but the budget is real: you pay for the slow side in GPU hours.

One more gap: AWS's thresholds page proposes scaling down when the queue is 0 and p95 is under 3 seconds for 5 minutes, but the ScaledObject encodes only the 5-minute window against the 25 and 5 targets. The 3-second condition is advice, not configuration.

What breaks first

These are the failure points the sources point at, ordered by how soon I would expect to meet them.

• A Ready replica is not a warm one. The warm-up page shows first-request latency falling from about 2.6 seconds to a steady baseline, so traffic sent to a fresh replica can trip your own p95 guardrail and trigger another scale up.
• No GPU node free. Karpenter has to provision one before the pod can even start loading weights, so the real time to capacity is node plus model load, far above the 30-second scale-up window.
• Scale-in mid-request. KEDA's page warns that the HPA can terminate a replica in the middle of a long-running task and suggests SIGTERM handling, a preStop hook or running the work as Jobs. For streaming completions that is a real concern.
• Prometheus goes quiet. The scaler's ignoreNullValues defaults to true, so an empty result is ignored; setting it to false makes the scaler error when the target is lost. KEDA's fallback section defines replicas to hold when a scaler is in an error state, and it is not supported for CPU and memory triggers.
• Copied thresholds. 25 and 5 came from one model on one instance type under one prompt shape. A larger model or a longer max_tokens moves the knee, and the guardrail then fires constantly or never.

llm-d moved its guides to KEDA

The llm-d v0.9.0 release notes carry no prose about autoscaling, only pull request titles, so I quote those and nothing more: convert EPP autoscaling guide to KEDA; migrate nightly and guide from Prometheus Adapter to KEDA; add KEDA support and shared-cluster setup; and authenticate KEDA to Thanos and assert the metric path. The notes list the workload variant autoscaler at v0.9.0 as well.

What I take from it, as an inference from titles: AWS's argument that KEDA reads Prometheus directly with no separate metrics adapter to manage is the same one a serving project made when it dropped Prometheus Adapter. The Thanos-auth change is a reminder that in a shared cluster the metric endpoint KEDA queries is usually not an open one, so budget for authentication in the trigger.

What to do on Monday

Do these in order. Each one replaces a number you would otherwise copy.

1. Run the warm-up, then the stepped load test against one replica of your own model and write down queue depth and p95 at each rate.
2. Set the queue threshold above the point where the queue first stays positive, and the latency threshold below your SLO.
3. Decide minReplicaCount on purpose, 1 or 0, and if 0, name the signal that wakes the first pod.
4. Confirm kubectl get hpa shows both metrics under TARGETS and read the events with kubectl describe.
5. Rehearse the bad days: stop the load and time the scale-down against your window, then break the Prometheus address and see what fallback does.

# My modification of the AWS ScaledObject (spec fields from KEDA's ScaledObject
# reference). NOT applied on a cluster by me. Only the changed parts are shown.
spec:
  minReplicaCount: 1        # set 0 only if something in front of vLLM can wake it
  maxReplicaCount: 5
  pollingInterval: 30       # KEDA default; how often each trigger is checked
  cooldownPeriod: 300       # default; applies ONLY to the 1 -> 0 step
  fallback:                 # replicas to hold when a scaler is in an error state
    failureThreshold: 3
    replicas: 2             # pick from your own load test; not a sourced value
  triggers:
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
        query: sum(vllm:num_requests_waiting) or vector(0)
        threshold: "25"          # REPLACE with the knee from YOUR load test
        activationThreshold: "1" # strictly greater than: needs 2+ waiting to activate
        ignoreNullValues: "false" # error (and fall back) when Prometheus is lost

# Rehearsal commands
kubectl get hpa vllm-inference-app -w
kubectl describe hpa vllm-inference-app | sed -n '/Events/,$p'
kubectl get pods -l app=vllm-inference-app -w

The fragment above is my modification of AWS's manifest using fields from KEDA's spec page, and I have not applied it. It keeps the same triggers, adds a fallback so a dead Prometheus holds capacity instead of guessing, and shows where scale to zero would be switched on.

Scale an inference server on what its users feel, the queue and the tail latency, scale up in seconds and down in minutes, and treat every threshold you did not measure yourself as a placeholder. The pages I read agree on the mechanism; the gaps are in the details they leave to you, like what activation actually gates and how long a GPU takes to become capacity.