vLLM Disaggregated Prefill Decode: A NixlConnector Guide

The most expensive mistake in a prefill and decode split is benchmarking one that is quietly broken. If the two GPUs cannot reach each other directly, the split still starts, still returns correct text and still draws a clean inter-token latency chart, while every request pays more than a second to move its KV cache. You find out when someone asks why time to first token tripled.

This post is a close reading of vLLM's own guide to disaggregated serving, published on 29 September 2026, alongside the NixlConnector documentation, the disaggregated prefilling feature page and the v0.31.0 release notes. I did not run the setup myself, so every measurement below is vLLM's and is attributed that way, and the commands are the ones those pages print. Where a figure is my own arithmetic rather than a measurement, I say so.

Why one vllm serve process fights itself

A single vllm serve process does three different jobs. Prefill reads the whole prompt in one compute-bound burst and decides time to first token. Decode generates one token at a time, is bound by memory bandwidth and decides inter-token latency. The CPU frontend does templating, tokenisation and output parsing. On one GPU the first two take turns on the same hardware, so a long prefill lands in the middle of everyone else's token stream.

The guide's measure of success is goodput: requests per second that meet both a time-to-first-token target and an inter-token-latency target, not peak tokens per second. The feature page is blunt about the other side of this: 'Disaggregated prefill DOES NOT improve throughput.' What it buys is the ability to tune the two latencies separately, with different parallel settings per instance, and to stop prefill interrupting decode. The same page marks the feature as experimental and subject to change.

The idea is older than vLLM's implementation. The DistServe paper assigns prefill and decoding to different GPUs and reports serving 7.4x more requests, or meeting a 12.6x tighter latency objective, than the systems it compared against. Treat that as a research ceiling rather than a forecast for your model. The rest of this post is about how much of it survives contact with a real interconnect.

Do the KV transfer arithmetic before you build anything

Every request in a split has to move its prompt's KV cache from the prefill GPU to the decode GPU before the first generated token. The size is fixed by the model, so you can estimate the bill from the model card. The guide puts Llama-3.1-70B in BF16 at 320 KiB of KV cache per token. A 10,000-token prompt is therefore about 3 GB, and at 400 Gb/s the wire time alone is about 64 ms, which is my arithmetic: 3.2 GB times 8 bits, divided by 400 Gb/s, before any protocol overhead.

The second row is the lesson. Under a sixth of the data took about twenty times longer, because the path between the GPUs was not peer to peer. I am comparing a theoretical wire time with a measured one on different hardware, so read that as an order of magnitude, not a ratio. The point is that bandwidth is not the only variable: the route is. The guide says to check two things before any benchmark.

# 1. Topology. On one box this must say OK between the prefill GPU and
#    the decode GPU. Anything else means KV moves by a slower path.
nvidia-smi topo -p2p r

# 2. Send a few long prompts ONE AT A TIME through the proxy, then read the
#    decode instance's log for its "KV Transfer metrics" line. If
#    "Avg xfer time" is in the hundreds of milliseconds, fix the transport
#    before you measure anything else.
grep "KV Transfer metrics" decode.log | tail -n 5

The first confirms the topology and the second reads what a real transfer cost. Neither needs a benchmark harness, and both take a minute. If either looks wrong, nothing you measure afterwards means anything.

Stand up the three processes

The guide's minimal deployment is two vllm serve instances and a proxy, and it assumes vLLM v0.31.0 or later. The prefiller takes kv_role kv_producer, the decoder takes kv_consumer, and both load NixlConnector. NIXL installs with uv pip install nixl on NVIDIA; other platforms build it from source against UCX. I replaced the guide's Qwen3-0.6B wiring example with Qwen2.5-7B-Instruct, because the guide says the 0.6B model prefills so fast that it barely disturbs decode and there is nothing for the split to fix.

# Requires vLLM v0.31.0 or later, and NIXL: uv pip install nixl

# Prefiller on GPU 0. kv_producer = this instance computes KV and hands it off.
CUDA_VISIBLE_DEVICES=0 UCX_NET_DEVICES=all VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}' \
  2>&1 | tee prefill.log

# Decoder on GPU 1. Its side-channel port MUST differ from the prefiller's
# whenever both run on one host: each worker needs a unique port per host.
CUDA_VISIBLE_DEVICES=1 UCX_NET_DEVICES=all VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}' \
  2>&1 | tee decode.log

# Development proxy from the vLLM repository. Clients talk to :8192.
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8192 \
  --prefiller-hosts localhost --prefiller-ports 8100 \
  --decoder-hosts localhost --decoder-ports 8200

The NixlConnector docs example also passes --enforce-eager. I left it out so a benchmark runs the engine the way production would. The proxy is toy_proxy_server.py from the vLLM repository, and the guide says it is for development only, naming llm-d and Dynamo as the production options. The settings that matter are these.

Across machines, give each instance its own VLLM_NIXL_SIDE_CHANNEL_HOST and hand the proxy every instance's address and port. For multi-host data parallel, the docs say only the head instances' host and port are needed.

What the proxy does to each request

The proxy is where the split becomes visible, so it is worth knowing its four moves. Clients talk only to its port, 8192 in the example, and never to prefill or decode directly.

1. The proxy sends the request to prefill with max_tokens set to 1 and kv_transfer_params containing do_remote_decode true.
2. Prefill computes the KV cache, holds the blocks and returns kv_transfer_params that point at them.
3. The proxy forwards the original request to decode with those params attached.
4. Decode pulls the blocks over NIXL, then generates.

# Step 1: what the proxy sends to the PREFILL instance (completions API).
# max_tokens is 1 because prefill only has to produce the KV cache.
{
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "prompt": "<the full long prompt>",
  "max_tokens": 1,
  "kv_transfer_params": {"do_remote_decode": true}
}

# Step 2: prefill answers with kv_transfer_params describing the blocks it is
# holding. Step 3: the proxy sends the ORIGINAL request to the DECODE instance
# with those params attached. Step 4 happens inside decode: it pulls the
# blocks over NIXL, then generates.

# Chat completions variant: call prefill with "return_token_ids": true, then
# give decode its kv_transfer_params plus "prompt_token_ids" from prefill.

For chat completions the guide's variant asks prefill for token IDs and hands them to decode. The feature page marks that token-ID reuse experimental: the IDs are ignored if the messages hold non-text content or echo is set, and invalid IDs give an HTTP 400. The practical consequence is that a proxy bug looks like a vLLM bug. Send one short request through the proxy and confirm a normal completion before you add any load.

Benchmark goodput, not tokens per second

The guide measures with vllm bench serve and a goodput definition of time to first token within 2000 ms and time per output token within 30 ms. Its workload is a random dataset of 8,192-token inputs with 256 output tokens and ignore-eos, so every request runs to length. If you reuse seeds across request rates, the guide says to disable prefix caching, otherwise later runs can be served from earlier runs' cache.

# Goodput = requests per second that meet BOTH targets: TTFT 2000 ms, TPOT 30 ms.
# Run it against the collocated server and against the proxy on :8192 with the
# same hardware, then compare. Repeat per --request-rate.
vllm bench serve --model Qwen/Qwen2.5-7B-Instruct --port 8192 \
  --dataset-name random --random-input-len 8192 --random-output-len 256 --ignore-eos \
  --request-rate 0.4 --num-prompts 100 \
  --percentile-metrics ttft,tpot,itl --metric-percentiles 50,99 \
  --goodput ttft:2000 tpot:30

Read the table for what it does not say. The median inter-token latency is the same in both setups, so the split does nothing for the typical token. It removes the tail: p99 at 0.4 req/s fell from 169 ms to 29 ms, and the guide says the split never exceeded 52 ms. The price is in the last row, where the L40S box had no peer-to-peer access and median time to first token went from 0.7 s to 2.2 s. A split only wins if your service-level target cares about the tail more than about first-token time.

Two external results are relayed by the guide, and I have not checked the underlying benchmarks. On a single 8xMI300X node running Qwen3-235B-A22B-FP8, AMD's MoRI-IO connector met both targets for 73 of 100 requests against 30 of 100 collocated, about 2.4x the goodput. llm-d reports about 59% lower mean end-to-end latency and 67% lower P95 for gpt-oss-120b on 16 H200s. Both use larger models on data-centre accelerators, not the L40S pair above, so do not read them as a prediction for a small box.

How a broken split passes for a working one

A split can be wrong in ways that still return text. These are the four failures the sources describe, with the check that exposes each. The first row is the one that bites a benchmark: the inter-token chart looks excellent while the first token crawls.

There is a deliberate choice in the failure policy. The default, fail, errors the request immediately and avoids recomputing prefill on the decode instance. The alternative, recompute, rebuilds the failed blocks locally on decode, which is how a request survives a failed transfer. The docs warn it can cause decode performance jitter and higher tail latency, and call it inefficient for production.

The multi-turn case deserves its own sentence. With bidirectional_kv_xfer set on both instances, prefill pulls earlier turns' blocks back from decode so only new tokens are computed. That is a real saving, and it is also why a template that rewrites history produces wrong output rather than slow output. The v0.31.0 release notes also list dead peer state being released without waiting for the TTL, which is one reason to run that version rather than an older one.

The GPU-less render tier, and what is still open

The second split moves CPU work, not GPU work. vllm launch render runs templating and parsing with no GPU, and vllm serve --tokens-only runs the engine on token IDs alone. Render, prefill and decode can then run as separate tiers, with your own code orchestrating the handoffs.

# CPU-only frontend: templating, tokenisation, parsing. No GPU needed.
# The parser flags must match the engine's configuration, or the split into
# content / reasoning / tool_calls will differ from an in-process server.
vllm launch render Qwen/Qwen3-0.6B --port 8100 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes

# Engine on token IDs only. Rejects stop strings; cannot return text itself.
vllm serve Qwen/Qwen3-0.6B --tokens-only --port 8200

# The request travels three hops:
#   POST :8100/v1/chat/completions/render     -> token IDs
#   POST :8200/inference/v1/generate          -> generated token IDs
#   POST :8100/v1/chat/completions/derender   -> text, reasoning, tool calls
# Wrong: calling derender with only the generated tokens. It needs
#   chat_request, the original request, passed back in. Without it: HTTP 400.

The guide's numbers are small, which is the appeal. Rendering a 9,000-token prompt costs about 15 ms of CPU, and one render server handled 73 requests per second on just over one core. The worker count is set with --renderer-num-workers and defaults to 1. The guide also warns that render and engine configurations must match, or the content, reasoning and tool_calls split will differ from an in-process server.

The open issues are the part to read before adopting it. Streaming derender with a parser replays the full token history on each chunk, which the guide measured at roughly 9x the in-process CPU, with a caching layer tracked in issue 57571. Long prompts are resent on every streamed chunk. Batch derender can miss finish_reason tool_calls (issue 47931), and logprobs come back as placeholder strings (issue 57574). I would treat 73 requests per second as an upper bound to measure against your own traffic.

What to do on Monday

Here is the order I would work in, stopping at the first step that fails. The checks are cheap and the benchmark is not, so they come first.

1. Confirm vLLM v0.31.0 or later on both hosts and install NIXL.
2. Run nvidia-smi topo -p2p r and require OK between the prefill and decode GPUs. On a box that cannot show OK, expect something like the 1.3 s transfer the guide measured.
3. Start prefill and decode with distinct side-channel ports and kv_load_failure_policy fail, then send one short request through the proxy.
4. Send a few long prompts one at a time and read Avg xfer time in decode's KV Transfer metrics line. Hundreds of milliseconds means fix the transport first.
5. Only then run the goodput benchmark, with a 7B-class model, against a collocated baseline on the same hardware, with prefix caching off if you reuse seeds.

#!/usr/bin/env bash
# preflight.sh: stop at the first check that fails.
set -euo pipefail

# 1. Version. The guide assumes v0.31.0 or later.
python -c "import vllm; print(vllm.__version__)"

# 2. Peer-to-peer between the GPUs. Read the matrix: you want OK between
#    the prefill and decode devices.
nvidia-smi topo -p2p r

# 3. One short request end to end through the proxy before any load.
curl -s localhost:8192/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-7B-Instruct","prompt":"Say hello.","max_tokens":16}'

# 4. A few long prompts at a low rate, then read the transfer cost.
vllm bench serve --model Qwen/Qwen2.5-7B-Instruct --port 8192 \
  --dataset-name random --random-input-len 8192 --random-output-len 64 \
  --ignore-eos --request-rate 0.1 --num-prompts 5
grep "KV Transfer metrics" decode.log | tail -n 5

# 5. Is the decode side losing blocks to lease expiry? (kv_lease_duration)
curl -s localhost:8200/metrics | grep nixl_num_kv_expired_reqs

Write the go or no-go rule down before you run the benchmark. If your p99 inter-token latency already meets its target collocated, the split costs you an extra deployment role and a transfer path for no gain, since the docs are explicit that it does not improve throughput. If the tail is what hurts, and the transfer check passes, the numbers above say it is worth a trial.

Disaggregation is a latency-shaping tool with a transfer bill, not a free speed-up. The rule to carry: measure the KV transfer path first, benchmark goodput against a collocated baseline on identical hardware, and keep kv_load_failure_policy on fail until the numbers are real.