vLLM NixlPushConnector: Push vs Pull KV Transfer and TTFT

The number that made me open the post was 1,744 ms. That is the time to first token vLLM reports for its default pull connector at 8 requests a second with 16k-token prompts, on a machine with 3.2 Tb/s of network. Nothing about that hardware looks slow, which means the time is going somewhere other than the wire.

This is a reading, not a benchmark. I did not run the connector. I went through vLLM's announcement of 14 October 2026, the push connector design document, the NixlConnector usage guide, an open pull request and the bidirectional transfer post, and I only quote numbers that I read in them. Where I interpret rather than quote, I say so.

Why KV transfer sits on the TTFT path

In a disaggregated deployment, prefill and decode run as separate vLLM instances, so the KV cache a prefill instance computes has to reach the decode instance before the first token can be generated. Transfer efficiency is therefore part of TTFT. vLLM's own docs give the reasons to split in the first place: each side can use its own parallel strategy, and tail inter-token latency is easier to control than with chunked prefill alone.

The default connector is pull-based. After prefill, the router relays the source metadata, meaning the prefill block IDs, to the decode instance. Decode then allocates destination blocks, prepares the transfer and issues a NIXL READ. The post's central observation is that these steps run one after another, after prefill has finished, on the critical path to the first token.

# Pull (default NixlConnector): everything after prefill is serial.
#
#   prefill compute --> router relays prefill block IDs to decode
#                   --> decode allocates destination blocks
#                   --> decode prepares the transfer (descriptors, inline)
#                   --> decode issues NIXL READ
#                   --> first token
#
# Push (NixlPushConnector): decode's side is done before prefill ends.
#
#   decode allocates blocks --> PUSH_REG notification to prefill   (early)
#   prefill compute ---------------------------------------------+
#                                    registration + prefill done -+--> NIXL WRITE
#                                                                 --> first token

Two cautions before the details. The vLLM docs state plainly that disaggregated prefill does not improve throughput, and that the feature is experimental and subject to change. The push connector is also three days old as I write this, so treat everything below as a design to evaluate, not a settled default.

What push changes: register early, write when ready

Push reorders the same work. When decode allocates blocks for a remote-prefill request, it immediately sends a registration to the prefill instance as a NIXL notification, carrying its identity and the reserved block locations. That happens before prefill finishes. The design doc calls the message PUSH_REG, with a msgpack-encoded payload holding the request ID, the decode engine, host, port and TP size, and the block IDs.

Registration and prefill completion can arrive in either order, and whichever comes second triggers the write. The doc describes the matching: an exact request ID match first, then a fallback that strips the trailing per-engine random suffix, because the proxy gives both legs the same X-Request-Id and that suffix is the only difference. The completion notification reuses pull mode's format, so decode's accounting code is unchanged.

The write goes straight into the decoder's registered memory, addressed per tensor-parallel rank. Each side maps blocks to its own layout, so prefill and decode block sizes may differ, and decode counts one completion notification per writing rank before it starts.

Where the time goes: descriptor work and the writer thread

The post blames a specific cost. In pull mode, NIXL descriptors are posted inline on the decode worker, synchronously inside its engine loop, so that work stalls token generation. That is also why the post reports a TPOT gain and not only a TTFT gain: the decode worker was doing transfer bookkeeping between tokens.

In push mode each prefill worker, one per TP rank, runs a thread named nixl-push-writer. It establishes or reuses the connection, maps block IDs to descriptors, merges contiguous blocks and submits the WRITE. The design doc says the thread sleeps on a threading.Event, is woken by new registrations or finished blocks, by every get_finished step and by handshake callbacks, and polls itself every 1.0 ms while finished blocks wait for a registration that has not arrived.

There is an open pull request, number 45211, that applies the same offload to pull mode by moving READ submission onto a nixl-pull-reader thread. Its author reports mean TTFT falling from 215.88 ms to 133.46 ms on Qwen3-32B with 2,048-token inputs at 8 QPS. As I read it, the thread offload is separable from the push protocol, so some of the push gain may be reachable without push. That is my inference, and the workloads differ: 2k inputs there, 16k here. The PR was still open and flagged needs-rebase when I read it.

Whether the remaining gap is registration overlap or threading is not something the published numbers can separate. The post gives no ablation, and I will not pretend it does.

The numbers vLLM published

The test used Qwen3-32B in a 1P1D topology on p5en.48xlarge nodes, each with 8 H200 GPUs and 16 EFA interfaces at 200 Gb/s, in one availability zone, on vllm 0.29.0 with nixl 1.4.1. Decode ran with prefix caching disabled so every request fetched its full KV cache, which is the heaviest case for the transfer path. The sweep covered 1k to 16k input tokens, 1 to 8 QPS and 128 output tokens.

Read the table for its shape, not only its headline. The gain grows with load and prompt length, and it is absent where the transfer is small. That fits an explanation based on serial work: a fixed amount of inline setup hurts more when requests queue behind it. The post publishes charts rather than tables, so every value above is the approximate figure stated in its text.

It is also one model on one hardware family, and the post itself says results depend on workload and configuration. Do not read 3x as a promise for your cluster.

Switching it on: the config that changes

The change is the connector class on both sides. The pull baseline below follows the post's launch command for the prefill node and the usage guide's kv_role values. For push, the post's example prefill configuration adds kv_parallel_size and selects the LIBFABRIC backend, the one its EFA setup uses.

# Pull baseline, prefill node. Shape follows the post's launch command.
# Set VLLM_NIXL_SIDE_CHANNEL_HOST to this node's IP when P and D are on
# different machines; the default is localhost and the port defaults to 5600.
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve Qwen/Qwen3-32B --served-model-name Qwen --dtype bfloat16 \
  --tensor-parallel-size 8 --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

# Pull baseline, decode node. The benchmark turned prefix caching off here
# "to force every request to fetch full KV from prefill".
vllm serve Qwen/Qwen3-32B --served-model-name Qwen --dtype bfloat16 \
  --tensor-parallel-size 8 --port 8200 --no-enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

# Push, prefill node: swap the class, and (as the post's example does)
# add kv_parallel_size and the LIBFABRIC backend used with EFA.
vllm serve Qwen/Qwen3-32B --served-model-name Qwen --dtype bfloat16 \
  --tensor-parallel-size 8 --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlPushConnector","kv_role":"kv_producer","kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"]}}'

# Push, decode node: same class swap, plus the PREFILL_* coordinates
# (engine id, host, side-channel port, TP and PP size). Take the exact
# JSON from the post's serve_pd.sh; I could not retrieve it.

# Wrong: NixlConnector on one node and NixlPushConnector on the other.
#        The post's symptom is a server crash right after the first request.
# Right: the same class on prefill and decode.

I could not retrieve the post's serve_pd.sh, because the asset URL returned a 404 page, so I have not seen the exact decode-side push JSON or how the prefill coordinates are passed. The post names them: PREFILL_ENGINE_ID, PREFILL_KV_HOST, PREFILL_SIDE_CHANNEL_PORT, PREFILL_TP_SIZE and PREFILL_PP_SIZE. Take the decode launch from that script rather than from my sketch.

What breaks: failures that decode only learns about by timeout

The design doc's failure section is the most useful part of this whole story, because it shows how push changes who notices a problem. In pull, decode drives the transfer and sees it fail. In push, prefill drives it, and several prefill-side failures never produce a failed receive on decode.

The watchdog timeout defaults to decoder_kv_blocks_ttl, and the usage guide gives that as 480 seconds by default. If both defaults hold, a lost registration can sit for eight minutes before it is dropped, so your client timeout is the real failure detector. Setting push_registration_timeout explicitly is the obvious lever; the doc names it but does not recommend a value, and neither will I.

There are also hard limits. Decode-side pipeline parallelism is unsupported because completions are counted per consumer rank, Mamba and SSM hybrids are unsupported under PP, and packed PP push requires MLA caches and equal prefill and decode block sizes.

When pull still wins

vLLM does not present push as a replacement. Its own guidance is to choose push when picking the decode instance early is acceptable and lower TTFT is the priority, and to keep pull in three situations.

• Load-aware routing. Pull lets the router choose the decode instance after prefill, using fresh load information. Push commits earlier, which is the price of overlapping setup with prefill.
• Bidirectional KV transfer. The usage guide enables it with bidirectional_kv_xfer on a NixlConnector deployment, and the post points to pull for it. The related post describes a prefill node issuing a D-to-P NIXL READ for previously computed turns, and reports up to about 1.6x lower TTFT on Qwen3-32B at 24k tokens. I found nothing saying push supports it.
• Anything the push limits exclude: decode-side pipeline parallelism, SSM hybrids under PP, or mismatched block sizes with packed PP.

These are different levers on TTFT. Push shortens the path the KV travels on each request. Bidirectional transfer avoids recomputing history on multi-turn conversations. A chat product with long shared context and a batch summariser with one-shot 16k prompts should not choose the same connector for the same reason.

What to do on Monday

If you run disaggregated vLLM on RDMA-capable networking, test push as a controlled change, not a rollout.

1. Confirm your workload resembles the test: long prompts and sustained load, with decode fetching most of the KV. If prompts are short and traffic is light, the post says to expect no difference.
2. Verify the fabric first. On AWS run fi_info -p efa, and confirm the NIXL side-channel port (5600 by default) is open between nodes.
3. Capture a pull baseline at your real prompt length with the script below before changing anything.
4. Flip the connector on both sides, keep the model, proxy and traffic identical, and compare TTFT and TPOT at several request rates, not one.
5. Set client timeouts and alerts for the silent failures above, and keep a pull configuration ready if you need load-aware routing.

# Step 2: is the fabric there, and is the NIXL side channel reachable?
fi_info -p efa | head
nc -zv "$PREFILL_IP" 5600

# Step 5: late notifications on either connector (name from the usage guide).
curl -s "http://$DECODE_IP:8200/metrics" | grep nixl_num_notifications_after_expiry

# probe_ttft.py - rough time to first streamed chunk through the toy proxy.
import json
import random
import time
import urllib.request

URL = "http://localhost:8192/v1/completions"  # toy proxy from the usage guide
MODEL = "Qwen"  # --served-model-name in the post's launch command
N_NUMBERS = 5000  # tune until usage.prompt_tokens is near 16k
RUNS = 5


def ttft_ms() -> float:
    # Random content: a repeated prompt would hit the prefill prefix cache
    # and flatter both modes.
    prompt = " ".join(str(random.randint(0, 99999)) for _ in range(N_NUMBERS))
    body = json.dumps(
        {"model": MODEL, "prompt": prompt, "max_tokens": 128, "stream": True}
    ).encode()
    req = urllib.request.Request(
        URL, data=body, headers={"Content-Type": "application/json"}
    )
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=120) as resp:
        for line in resp:
            if line.startswith(b"data: ") and b"[DONE]" not in line:
                return (time.perf_counter() - start) * 1000
    raise RuntimeError("stream ended without a data chunk")


samples = sorted(ttft_ms() for _ in range(RUNS))
print("TTFT ms:", [round(s) for s in samples], "median:", round(samples[RUNS // 2]))

The script is a rough probe of time to first streamed chunk through the toy proxy from the usage guide, which listens on 8192 in its example. I have only syntax-checked it. Random numbers avoid prefix-cache hits on repeat requests, and you should tune the word count until a non-streaming request reports about 16k prompt tokens.

Push KV transfer is a scheduling change more than a networking one: tell the producer where the data goes before the data exists, and the transfer stops queueing behind prefill. Measure at your own prompt lengths and rates, keep pull for load-aware routing and bidirectional transfer, and treat a quiet prefill-side failure as something only a timeout will catch.