vLLM 0.31 Release Notes: Preload, Model Runner V2 and the Flags That Break

I went through the release pages for vLLM v0.29.0, v0.30.0 and v0.31.0 looking for one answer: what changes for someone who restarts a vLLM server in production. The answer is spread over three releases in four weeks. The daemon appeared in 0.30, became a proper command in 0.31, and the runner it sits on switched default in 0.29. Read only the newest page and you miss two thirds of the story.

My sources are the three GitHub release pages, the v0.31.0 preload and CLI docs, the initialized-snapshots docs, one third-party upgrade checklist, and LMSYS's write-up of the same idea in SGLang. I did not run any of it on a GPU, so none of the numbers below are my measurements. Every flag, default and limit is quoted from a page I read, and where a source is secondary or the docs are not pinned to 0.31, I say so.

Three releases in four weeks, and the restart story spans all of them

Start with the timeline, because the breaking changes accumulate across it. The 0.30 notes describe a per-GPU weight-cache daemon that serves restarts through --load-format ipc_cache. The 0.31 notes promote it to a vllm preload CLI and add data parallelism, MTP draft models, a /health endpoint and a readiness wait. If you are upgrading from before 0.29, you inherit every removal in the table below, not just the latest page's list.

The release pages count commits and contributors themselves, and I have copied those counts as written. They are a reminder that this is not a patch cadence: each of these is several hundred commits, and each removed or renamed something an operator might have in a systemd unit. One honest limit: the v0.29 page does not print a year, so the year comes from the surrounding release dates, not from the page.

Model Runner V2 is the default, and V1 now has a removal date

The 0.29 notes state that Model Runner V2 is now the default for all models. They also say Model Runner V1 is deprecated, with removal targeted for v0.32, and that the project does not intend to accept more V1-specific optimisations. At that point V2 still lacked sequence parallelism, dual-batch overlap, elastic expert parallelism, custom logits processors and some speculative decoding methods, and vLLM fell back to V1 when you configured any of them.

The gaps are closing on schedule. The 0.31 highlights list draft-model speculative decoding and custom logits processors working on Model Runner V2, and 0.31 also rejects Model Runner V1 with pipeline parallelism above 1, async scheduling and structured output at startup. In 0.30, the all Mamba cache mode was deprecated and falls back to V1. The practical reading: a config that quietly ran on V1 in 0.29 now has a deadline, and one particular combination fails fast instead of limping.

Do not confuse this with startup cost. The 0.30 notes credit gc freezing in Model Runner V2 with cutting engine init from 28.9 s to 8.2 s on an H200, which is the project's own figure for its own setup. That is initialisation, not weight loading, and it is a different slice of a restart than the one preload attacks. I found no flag in the notes that reports which runner was chosen, so check your startup logs rather than assuming.

How the weight-cache daemon works, and what it keeps alive

vllm preload starts one daemon per GPU. Each daemon loads its shard, runs the quantization post-processing, and exports every parameter and buffer as a CUDA IPC handle over a Unix domain socket. An engine started with --load-format ipc_cache builds its model on the meta device, asks the daemon on its GPU for the tensors and maps them zero-copy, so a restarted engine never reads the checkpoint from disk. It works on CUDA and ROCm only.

# Step 1: one daemon per GPU. It loads each shard from disk ONCE, runs the
# quantization post-processing, and exports every tensor as a CUDA IPC handle.
# The command reports ready only when every rank is serving; if a rank dies at
# startup the others are terminated and the exit code is that rank's.
vllm preload --model meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4

# Step 2: engines map the daemon's tensors instead of reading the checkpoint.
# TP size, dtype, quantization, revision and the vLLM version must match the
# daemon's, or the engine quietly loads from disk (fallback defaults to true).
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 4 \
  --load-format ipc_cache

# Stop the daemon with SIGINT or SIGTERM: it removes its sockets and frees GPU
# memory. In the default zero_copy mode the engines must be gone first.

Before serving, the engine and daemon compare fingerprints covering checkpoint content hashed from safetensors metadata, architecture, TP and DP size and rank, dtype, quantization, model revision and the vLLM version. The engine also checks that the daemon's GPU UUID matches its own device, so a stale socket cannot hand it the wrong GPU. If the daemon is missing or any fingerprint differs, the engine loads from disk, and that is the default behaviour.

The idea is not new to vLLM. LMSYS published the same design for SGLang in August 2026: one daemon per GPU exporting CUDA IPC handles. On Ling-2.6-1T FP8 across eight H20-3e GPUs they report weight loading falling from about 495 s to about 0.63 s, and they say disk loading was 93.2 percent of startup time. Their own post is inconsistent about the disk figure, quoting other values elsewhere, so read it as order of magnitude. That is a different engine and model, not a vLLM measurement, and the post notes the first daemon start still reads from disk.

The loader config: two modes, one fallback, and the timeouts

The loader is tuned through --model-loader-extra-config on the engine. The mode decides who owns the weights: in zero_copy the engine maps the daemon's allocations and the daemon must outlive it, with one copy per GPU; in copy the engine clones every tensor into its own memory and the daemon releases its cache, which frees GPU memory but costs a full copy per restart. In zero_copy, sleep mode weight offloading must not be used with this loader.

Scaling out follows the node. IPC handles are node-local, so multi-node tensor parallelism needs one launcher per node, each serving only its local GPUs, with --weight-cache-master-port pinned and different from the engine's --master-port. With data parallelism, local GPU i serves DP rank start_rank plus i divided by tp_size, rounded down, and TP rank i modulo tp_size. For MTP, EAGLE and EAGLE3, a separate draft daemon group caches the draft model on its own socket names and rendezvous port.

# Strict engine config for rollout tests: fail instead of falling back to disk.
# connect_timeout_s and state_timeout_s are shown at their documented defaults.
vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4 \
  --load-format ipc_cache \
  --model-loader-extra-config '{"mode": "zero_copy", "fallback": false, "connect_timeout_s": 5.0, "state_timeout_s": 300.0}'

# Multi-node TP: one launcher per node, each serving only its local GPUs.
# The daemon's rendezvous port must differ from the engine's --master-port
# and match across nodes.
vllm preload --model /path/to/model --tensor-parallel-size 16 \
  --nnodes 2 --node-rank 0 --master-addr 10.0.0.1 \
  --weight-cache-master-port 29600
# node 1: the same command with --node-rank 1

Preload, snapshots, sleep mode: which restart tool fits

0.31 gives you two new ways to avoid a cold start, and they are not interchangeable. vllm snapshot create and vllm snapshot restore use CRIU to restore a fully initialised TP1 engine, which targets engine initialisation, although the docs say restore still reloads model files and KV cache before it binds HTTP. The cost is a much narrower box: x86-64 Linux, one NVIDIA GPU, root or passwordless sudo, io_uring disabled, a pinned 40-character model revision and a plaintext, unauthenticated HTTP server. I read the snapshot page at its latest URL, because the versioned 0.31.0 URL returned 404, so treat its limits as current docs rather than pinned to 0.31.

My rule of thumb from the table, and it is a judgement rather than a documented recommendation: preload is the one to try first, because its requirements are an environment most GPU hosts already meet, and it covers tensor, expert and data parallelism. Snapshots are for a single-GPU, single-model service where init time dominates and you can accept root, a privileged container and a plaintext listener inside a trusted boundary. Restore fails rather than falling back to a normal start, so it needs its own recovery path.

The breaking changes, in the order your deploy will hit them

Most of these fail loudly, which is the good case. The ones to read twice are the multimodal kwargs gate, which is a security change presented as a flag, and the GPTQ and YaRN rows, which change behaviour instead of refusing to start.

The script below greps a deploy repository for those strings plus three env vars the notes removed (VLLM_PREFIX_CACHE_RETENTION_INTERVAL, VLLM_MM_HASHER_ALGORITHM and VLLM_PLE_CPU_OFFLOAD). Two new knobs are worth knowing while you are in there: --max-num-active-seqs caps RUNNING admission separately from max_num_seqs (0.31), and --max-num-queued-reqs and --max-num-queued-tokens add admission control on the waiting queue (0.29).

# Run from the repo that holds your launch scripts, unit files and compose files.
# Each pattern is something the 0.29 to 0.31 notes removed, renamed or gated.
# A hit is a place to read, not proof of a bug: the quantization pattern also
# matches a deliberately spelled fp8 that now redirects to fp8_per_tensor.
grep -rnE \
  -e 'tokenizer[_-]mode[= ]+"?slow' \
  -e 'quantization[= ]+"?fp8"?( |$)' \
  -e 'enable-mamba-fine-grained-prefix-cache' \
  -e 'collect-detailed-traces[= ][^ ]*,' \
  -e 'VLLM_ENABLE_SCALE_OUT_ENDPOINTS' \
  -e 'VLLM_PREFIX_CACHE_RETENTION_INTERVAL' \
  -e 'VLLM_MM_HASHER_ALGORITHM' \
  -e 'VLLM_PLE_CPU_OFFLOAD' \
  -e 'vllm\.entrypoints\.openai\.api_server' \
  -e 'mm_processor_kwargs|media_io_kwargs' \
  --include='*.sh' --include='*.service' --include='*.yml' \
  --include='*.yaml' --include='*.py' .

Where it goes wrong: a start that succeeds is not a cache hit

The named failure is silent fallback. fallback defaults to true, so a daemon that is missing, on the wrong socket, or fingerprinted for a different vLLM version produces an engine that starts, passes its health check and loads from disk at the old speed. The third-party upgrade checklist puts it plainly: successful startup alone cannot prove a cache hit. Nothing I read exposes a metric for hit versus fallback, so the proof is to run the engine once with fallback false and see that it starts.

The second trade-off is that the version is part of the fingerprint. The docs say daemon and engine must use the same vLLM version, and the checklist says a 0.30 daemon is not a warm cache for a 0.31 engine. My inference, not a documented procedure: an upgrade is still a cold start, because the new daemon has to read the checkpoint from disk once before restarts get fast. Preload speeds up restarts of the same build, not upgrades.

Other limits are plainer. The daemon must stay alive for the engine's lifetime in zero_copy mode, and stopping it with SIGINT or SIGTERM removes the sockets and frees GPU memory. Quantization methods that cannot supply pre-processed weights raise an error, pipeline parallelism is rejected at launch, and the same-node, same-user rule means a Kubernetes pod boundary can split a daemon from its engine. LMSYS's SGLang version limits itself to unquantized and block-wise FP8, so check your quantization method against the vLLM docs rather than assuming parity.

What to do on Monday

Treat the upgrade and the cached restart as two separate changes, and ship them one at a time. The order I would use:

1. Run the grep audit above on every repository that launches vLLM, and fix each hit on the old version where the new spelling already works.
2. Upgrade without preload and check startup logs for which runner was chosen; if a config fell back to Model Runner V1, schedule that work before v0.32.
3. Audit multimodal clients for per-request mm_processor_kwargs and media_io_kwargs before deciding whether --trust-request-mm-kwargs is ever needed.
4. Start a preload daemon on one canary host and bring the engine up with fallback false, so a mismatch fails loudly.
5. Time a cold start, a restart through the daemon, and a deliberate mismatch on your own model and GPUs. Record the numbers yourself; no source I read gives you yours.
6. Only then set fallback back to true for production, and keep the previous container image and launch config for rollback.

#!/usr/bin/env bash
# Canary rollout of the weight-cache daemon. Defaults for socket locations on purpose.
set -euo pipefail
MODEL=meta-llama/Llama-3.1-8B-Instruct
TP=4

# Daemon first. /health is off unless a port is given; bind it to loopback.
vllm preload --model "$MODEL" --tensor-parallel-size "$TP" \
  --weight-cache-health-port 8100 --weight-cache-health-host 127.0.0.1 &

# The parent answers once the daemon is up. It does not prove the engine will
# hit the cache, and it runs no inference.
until curl -fs http://127.0.0.1:8100/health > /dev/null; do sleep 2; done

# Wrong: a plain engine start. It succeeds on a cache miss too, so a version
# or dtype mismatch is invisible.
#   vllm serve "$MODEL" --tensor-parallel-size "$TP" --load-format ipc_cache
# Right for the canary: fallback false turns a mismatch into a startup failure.
vllm serve "$MODEL" --tensor-parallel-size "$TP" \
  --load-format ipc_cache \
  --model-loader-extra-config '{"fallback": false}'

The rollout script above uses the defaults for socket locations on purpose. The docs derive a per-user private directory and per-GPU socket names, and overriding them is a separate decision you should only make if a launcher and an engine really run as different paths.

The rule I would carry out of these three releases: a faster restart is a new failure mode as well as a feature, because the fast path and the slow path both end in a running server. Make the fast path prove itself with fallback off, check the removed flags before the code, and measure on your own hardware, because the only vLLM numbers on the release pages belong to someone else's.