vLLM Bi-Directional KV Transfer for Multi-Turn Agents

Every agent loop I have read about ends the same way: turn after turn, the client sends the whole conversation again, tool calls and results included, and the server prefills all of it before it emits one token. On 12 October 2026 the vLLM team published a post about getting that repeated work back, and the detail that made me read it twice was where the cache lives. The decode node already holds the KV blocks for the earlier turns, including the tokens it generated. The prefill node, which does the expensive work, cannot see them.

This is a reading of primary sources, not a benchmark run of mine. I read the vLLM post, the NixlConnector and disaggregated prefill pages in the vLLM docs, the merged pull request and the RFC behind the feature, the still-open pull request the post's reproduction depends on, and AWS's EFA documentation. Every number below belongs to the vLLM team or its contributors and is attributed as such. Anything I could not find in a source, I left out.

Why the tenth agent turn pays for the first nine again

In a multi-turn agent, turn N sends turns 1 to N-1 again plus whatever is new. On a single node with prefix caching this is cheap, because the history is already there. In prefill-decode disaggregation it often is not: the post describes the prefill node recomputing the context whenever it misses its own cache, and the cost grows with the conversation. The post's own baseline shows it. With the feature off, Qwen3-32B's time to first token (TTFT) goes from 158 ms at a prompt of about 2k tokens to 778 ms at about 24k, and GLM-5-FP8's goes from 500 ms at about 2k to 2,334 ms at about 20k.

Divide those endpoints and the shape is clear. Prompts grew about 12 times for Qwen3-32B and 10 times for GLM-5-FP8, and TTFT grew about 4.9 and 4.7 times. That is my arithmetic on the post's endpoint figures, not a number the post states, and it mixes recompute with fixed overhead, so it is not a clean law. The point is narrower: an agent on turn forty pays a visible latency tax on every turn for text the cluster has already computed once.

Disaggregation does not fix this by itself. The vLLM docs say disaggregated prefill exists to tune TTFT and inter-token latency separately, by giving each instance its own parallel settings, and they state that it does not improve throughput. The page also marks the feature experimental. So the first question is not how to enable bi-directional transfer. It is whether your traffic is multi-turn and prefill-heavy enough that the repeated prefix is the thing you are paying for.

What one-way prefill-decode disaggregation leaves behind

In the standard flow, KV moves in one direction. The prefill node computes the prompt's KV and ships it to the decode node, which generates the answer. The decode node therefore ends up holding the KV for the prompt and for every token it generated. On the next turn the client resends all of that as input, and the prefill node, which never saw the generated tokens, has nothing to match them against. The RFC behind the feature names the two cases this creates: multi-turn conversations, where earlier model responses live only on decode, and evicted prefill cache, where session state sits on decode while prefill misses its own cache.

The RFC's author considered the obvious alternative, incremental prefill on the decode node, and rejected it because it defeats the purpose of disaggregation by making decode nodes run both paths. Pulling blocks back keeps the roles intact and changes only what moves over the wire.

The six steps of a decode-to-prefill pull

The vLLM post describes the flow in six steps. Reading them next to the RFC, the design choice that matters is that nothing queries the decode node on the hot path.

1. Decode retains the KV for the prompt and for its own response after the request finishes.
2. It returns the block metadata in the terminal chunk of the streaming response.
3. The router caches that metadata under the conversation_id.
4. On the next turn, prefill does its normal local prefix-cache lookup first, then checks whether remote reuse is possible.
5. If the reusable amount meets the threshold, it queues an asynchronous receive.
6. It loads the KV with a NIXL READ from decode to prefill and computes only the remaining suffix locally.

The RFC explains why the metadata rides on the response stream. Its author first proposed a separate query API to ask decode for its KV parameters, benchmarked it, and found the extra round trip added latency, so decode now returns kv_transfer_params in its streaming response. The query API remains only for initial node pairing. The router stores block locations, never KV tensors.

# My sketch of the router-side state the post describes. NOT vLLM's code.
import time

DECODER_TTL_S = 480   # decoder_kv_blocks_ttl on the decode node
ROUTER_TTL_S = 450    # shorter on purpose: stop asking before decode drops blocks

_cache: dict[str, tuple[dict, float]] = {}  # conversation_id -> (kv_params, stored_at)

def remember(conversation_id: str, kv_params: dict) -> None:
    # kv_params comes from the terminal chunk of decode's streaming response.
    # Block LOCATIONS only. The router never touches KV tensors.
    _cache[conversation_id] = (kv_params, time.monotonic())

def lookup(conversation_id: str) -> dict | None:
    hit = _cache.get(conversation_id)
    if hit is None:
        return None                      # first turn, or router restarted: full prefill
    kv_params, stored_at = hit
    if time.monotonic() - stored_at > ROUTER_TTL_S:
        del _cache[conversation_id]
        return None                      # decode may have dropped them: full prefill
    return kv_params                     # handed to prefill, which pulls via NIXL READ

That sketch is mine and deliberately naive. The real implementation is disagg_proxy_multiturn.py in the vLLM repository, which I have not read line by line. The docs add one detail worth knowing: conversation_id is a proxy extension and is not forwarded to vLLM itself, so your gateway has to mint it and carry it across turns, and the pull request that adds kv_transfer_params to streaming responses is a hard dependency.

Two knobs decide whether a pull happens: TTL and threshold

Two settings decide whether a pull happens and whether it is worth it. Both live in kv_connector_extra_config, and the post and docs give the defaults below.

A naming trap: the merged pull request's example config calls the threshold remote_pull_threshold, while the current docs and the October post call it kv_recompute_threshold. Check the key name against the version you run. I could not tell from the sources whether an unrecognised key raises an error or is ignored, so test it rather than assume.

Reading the numbers: 1.6x and 3.3x, and what they were measured on

Here are the figures as the post gives them. It prints endpoints in prose rather than per-length tables, so I reproduce only what it states.

Read the conditions before the ratios. One prefill node and one decode node, each a p5en.48xlarge with eight H200 GPUs, in the same availability zone, over EFA, with prefix caching deliberately disabled on the prefill node to emulate one that has evicted its local cache. The benchmark client, the conversation count and the concurrency are not stated. Dividing the Qwen3-32B endpoints gives about 1.7x rather than the 1.6x headline, so treat the headline as rounded. The post attributes the bigger GLM-5-FP8 gain to its higher per-token recompute cost; the transfer time sits in the same tens-of-milliseconds band for both models, so the model that is slower to recompute gains more. The post itself says results depend on the workload and deployment configuration.

The pull request is more candid about the case most production clusters are in. With prefix caching enabled on prefill, its author reports median TTFT falling from 307.86 ms to 219.61 ms, about 29 per cent, on Llama-3.3-70B on p5en. That is a good result and a very different one from 3.3x. The large ratios describe prefill nodes that miss their own cache; if yours rarely do, expect the smaller number.

Wiring it up: the config the vLLM post prints, and what it omits

The post prints the prefill side of the setup in full. This is that command and config as printed, with comments of mine on what each part is for.

# Prefill node, as printed in the vLLM post (Qwen3-32B, 8x H200, EFA).
# The post prints bidirectional_kv_xfer as false for the OFF leg; true is the ON leg.
export FI_PROVIDER=efa                      # EFA via libfabric; without it NIXL may use TCP
export VLLM_NIXL_SIDE_CHANNEL_HOST=<P_IP> VLLM_NIXL_SIDE_CHANNEL_PORT=5600

# --no-enable-prefix-caching goes on the PREFILL node only. It emulates a prefill
# node that has evicted its cache, which is the post's best case. Decode keeps it on.
vllm serve Qwen/Qwen3-32B --served-model-name Qwen --dtype bfloat16 \
  --tensor-parallel-size 8 --max-model-len 32768 --max-num-seqs 8 \
  --no-enable-prefix-caching --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_buffer_device":"cuda",
    "kv_role":"kv_producer","engine_id":"prefill-engine-001","kv_rank":0,
    "kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"],
    "kv_recompute_threshold":64,"decoder_kv_blocks_ttl":480,
    "bidirectional_kv_xfer":true}}'

The decode side is not printed. The post says to run serve_pd_multiturn.sh with ROLE=decode and to keep BIDIR identical on both nodes. The docs supply the roles: prefill is kv_producer, decode is kv_consumer, and the older kv_both is deprecated for NixlConnector. Copy the decode values from the repro script, not from my guess.

# Wrong: flag set on one side only. Nothing errors, the proxy logs "cache MISS"
# on every turn, and the run looks healthy while showing no speed-up.
#   prefill: "bidirectional_kv_xfer": true
#   decode:  "bidirectional_kv_xfer": false

# Right: identical on both nodes (the post's scripts take BIDIR=true on each),
# and restart the proxy between the OFF and ON legs: its cache is in-process.

# The multi-turn proxy takes hosts and ports, not URLs.
python3 $VLLM_SRC/examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
  --port 8000 --prefiller-host <P_IP> --prefiller-ports 8100 \
  --decoder-hosts <D_IP> --decoder-ports 8200

# Before reading a single TTFT number, prove the transport on BOTH nodes:
fi_info -p efa | head -n 5        # empty output = no EFA provider = invalid run

# Then watch for the two log lines that explain most bad runs:
#   "Declining expired remote read"   -> TTL lapsed before the next turn
#   "KV Transfer metrics:"            -> transfer timings (prefill node log)

The transport is part of the result. AWS documents that EFA integrates with libfabric and supports NIXL, that NIXL needs libfabric 1.21.0 or later, that RDMA read is supported on instances with Nitro version 4 and later, and that EFA traffic cannot cross availability zones or VPCs. That last limit is why the post's two nodes share a zone. Without EFA, the post warns, NIXL either fails or falls back to TCP sockets and the transfer numbers are invalid. The vLLM docs add that the default NIXL backend is UCX, and that bi-directional mode is currently CUDA-only with a device-buffer KV cache.

Where it breaks: expiry, router state, reasoning traces

Five failure modes appear in the sources, and none is exotic.

• TTL expiry. The post says the TTL is fixed and not renewed, and that a turn arriving later falls back to normal prefill. An agent that waits on a slow tool or a human approval for longer than 480 seconds loses its cache. The post suggests a heartbeat-renewed lease as future work.
• Router state lives in memory. The post notes the proxy's conversation cache is in-process, which is why it must restart between the off and on runs. My inference, not the post's: a router restart, or a second router replica, starts with an empty map and every conversation pays a full prefill once.
• Reasoning traces. The docs warn that reasoning models which strip their thinking traces between turns can produce incorrect results, and say the router must detect the mismatch.
• Trust. In the RFC thread, a maintainer flagged that having clients return kv_transfer_params on later turns is risky because a malicious client could exploit it, and suggested tracking sessions in the router or load balancer instead.
• A dependency still in review. The post's reproduction applies pull request 40186 to both nodes. When I read the page on 17 October 2026 it showed the PR as open, with a request for re-approval after a rebase on 1 October.

On maturity: the merged pull request landed on 30 April 2026, the RFC was closed as completed on 15 June 2026, the docs now describe the feature, and the post says it was first built and validated on AWS Trainium before being contributed to vLLM. That is a real trail, but the benchmarks it rests on cover one workload per model and one transport. I would call it early.

What to do on Monday

Do not start by enabling the flag. Start by finding out whether your traffic has the problem at all.

1. Log conversation_id, a timestamp and the prompt token count for every request, then run the script below to see how much of each prompt was already in the previous turn and how many turns arrive after the TTL.
2. If the repeated share is small, or most gaps exceed 480 seconds, stop. Prefix caching on a colocated node is the simpler answer, and that is my judgement, not a vLLM claim.
3. If you do disaggregate, run the A/B with prefix caching enabled on prefill, because that is your production state, and keep the disabled run as the best case.
4. Verify the transport before you read any TTFT: fi_info -p efa must list an EFA provider on both nodes.
5. Set the router TTL below decoder_kv_blocks_ttl, and alert if the cache MISS rate sits at 100 per cent, which the post lists as the symptom of a misconfigured run.

#!/usr/bin/env python3
# usage: python3 reprefill.py < requests.jsonl
# one JSON object per line: {"conversation_id": "c1", "ts": 1760000000.0, "prompt_tokens": 18234}
import collections
import json
import sys

TTL_S = 480  # match decoder_kv_blocks_ttl

convs = collections.defaultdict(list)
for line in sys.stdin:
    r = json.loads(line)
    convs[r["conversation_id"]].append((r["ts"], r["prompt_tokens"]))

prompt_total = repeated = turns = late = 0
for rows in convs.values():
    rows.sort()
    for (prev_ts, prev_tok), (ts, tok) in zip(rows, rows[1:]):
        turns += 1
        prompt_total += tok
        # Lower bound: the previous PROMPT is certainly part of this prompt.
        # The previous response is too, but we do not log it, so we ignore it.
        repeated += min(prev_tok, tok)
        if ts - prev_ts > TTL_S:
            late += 1

print(f"follow-up turns:            {turns}")
print(f"repeated prompt share (>=): {repeated / prompt_total:.1%}")
print(f"turns after the TTL:        {late / turns:.1%}")

The script reports a lower bound, because it counts only the previous turn's prompt tokens as repeated and ignores the response tokens that decode also holds. I would rather under-claim the opportunity than size a cluster change on an optimistic estimate.

The rule I take from this: in disaggregated serving, the cache you need may be on the other node, and the question is whether pulling it back beats recomputing it. Pulling wins when prompts are long, turns arrive close together and the transport is RDMA-class. The vLLM team's 3.3x is real for the case it measured, about 29 per cent is the number to expect when prefill's own cache is healthy, and the feature is young enough that you should read the open pull request before you build on it.