LMCache Multi-Process Mode: Shared KV Cache for MoE Serving

The number that made me open this post was in its title: a 10x improvement for mixture-of-experts inference. A figure that round usually hides a specific denominator, so I read the LMCache write-up, the multi-process documentation behind it and vLLM's data parallel guide side by side before forming an opinion.

I did not run this setup; an 8x H100 node is not on my desk. Everything below is a close reading of five primary sources, with the arithmetic redone and the caveats the headline leaves out. Each figure is attributed to the page I read it on, and anything those pages do not say is marked as unknown rather than guessed.

Why DP plus EP leaves each rank with its own KV cache

The baseline in the post is vLLM with data parallelism for attention and expert parallelism for the experts. vLLM's data parallel guide describes each DP rank as a separate core engine process that talks to the front end over ZMQ sockets, and states that each DP engine has an independent KV cache. Weights for attention are replicated per rank; the experts are sharded across them once expert parallelism is enabled.

That layout is good for throughput and bad for reuse. The LMCache post says each request is pinned to one DP rank, so a multi-turn conversation only benefits from a cached prefix if its next turn lands on the same rank that served the previous one. The vLLM guide says its internal load balancer uses each engine's running and waiting queues, that KV-cache-aware routing is not yet implemented, and that prefix caching gains depend on directing prompts sensibly without saying how.

The consequence is fragmentation. Even with CPU offload enabled, the post says caches are not shared between ranks, so overlapping prefixes seen on different ranks are recomputed. For chat, agents and RAG, where each turn extends the last, that is repeated prefill work you pay for in time-to-first-token. Note this is a claim about mechanism; the post reports no cache hit rates, so how often it happened in the test is not measured.

The baseline: in-process offload, one pool per rank

The in-process baseline uses vLLM's native offloading with 50 GB of host memory per rank. Across eight ranks that adds up to 400 GB, and the launch looks like this.

# Baseline from the LMCache post: 8-way DP, experts sharded by EP,
# and one isolated 50 GB host-memory offload pool PER RANK (8 x 50 = 400 GB).
VLLM_USE_FLASHINFER_MOE_FP8=0 VLLM_USE_DEEP_GEMM=1 \
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.8 \
  --max-num-batched-tokens 1024 \
  --kv-offloading-size 50 \
  --disable-hybrid-kv-cache-manager \
  --max-model-len auto

# Each rank can only reuse prefixes that rank itself has seen. A follow-up turn
# routed to a different rank than its first turn finds nothing and recomputes.

Two details matter later. First, the post notes that the disable-hybrid-kv-cache-manager flag is required for hybrid-memory-allocator models with the native offloading connector. Second, the 400 GB total is the same number the multi-process run uses for its single pool, so the comparison gives both sides equal host memory. What differs is whether the memory is one pool or eight.

What multi-process mode changes architecturally

In multi-process mode LMCache stops being a library inside each vLLM process and becomes a standalone service. The documentation says engines connect to it through a ZMQ or gRPC request transport, and the README describes it as a daemon that runs separately from the inference engine. The comparison below is assembled from the post, the multi-process docs and the README.

Inside the server, the docs describe a StorageManager that coordinates the L1 pool and optional L2 adapters, with controllers for storing to L2, evicting from L1 and prefetching L2 hits back into L1. Eviction policies listed are LRU, ARC, IsolatedLRU and noop. The post's run uses LRU. L1 defaults to pinned CPU DRAM, with Device-DAX and an NVMe slab through GPUDirect Storage as alternatives.

The configuration that produced the result

The post publishes both launch lines, which is what makes it checkable. The MP run starts a server with a 400 GB L1 pool and LRU eviction, then launches the same DP plus EP vLLM command with a KV connector instead of the offload flags.

# Terminal 1: the LMCache server owns ONE host-memory L1 pool for the node.
# 400 GB is the pool the post used on its 8x H100 box.
lmcache server --l1-size-gb 400 --eviction-policy LRU

# Terminal 2: the same vLLM DP+EP launch, with the connector instead of
# --kv-offloading-size. The post also drops --disable-hybrid-kv-cache-manager
# and does not say why, so treat that as a second variable.
VLLM_USE_FLASHINFER_MOE_FP8=0 VLLM_USE_DEEP_GEMM=1 \
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.8 \
  --max-num-batched-tokens 1024 \
  --max-model-len auto \
  --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both"}'

# The post's load generator: multi-round chat at 2.0 QPS for 120 seconds.
lmcache bench engine --engine-url http://localhost:8000 \
  --workload multi-round-chat --mrc-qps 2.0 --mrc-duration 120

Read the difference carefully. The MP command has no kv-offloading-size and no disable-hybrid-kv-cache-manager, and the post does not explain either omission. So the two runs differ in more than the connector, and the published delta cannot be attributed to shared memory alone. If you reproduce this, change one thing at a time.

Reading the benchmark: what 10x actually is

The test used Qwen3-235B-A22B-Instruct-2507-FP8 on one server with eight H100 80 GB GPUs, vLLM 0.18.1 and LMCache 0.4.3-dev, driven by a multi-round chat workload at 2.0 QPS for 120 seconds. These are the post's table values with the ratios recomputed.

The headline 10x matches the p99 time-to-first-token ratio, not the mean. The mean ratio is larger, and the post's bar chart labels it 13.6x, close to my 13.7x from the rounded table values. The decode figures are smaller gains, and the post does not define whether decode speed is per request or aggregate, so I would not quote the 3.8x without that caveat.

The post credits higher cache hit rates, less redundant prefill and less fragmentation, but shows no hit-rate data to confirm the split. It is one model, one machine, one workload and one QPS level, with no repetitions or variance, on a development build of LMCache. The conversation count and prompt and output lengths are not stated. Treat it as a strong existence proof for the mechanism, not a forecast for your cluster.

The trade-offs and where it can bite

A shared cache service is one more moving part on the hot path of every request. From the documentation, these are the constraints I would plan around before adopting it.

• Version lockstep: the docs say to upgrade the server and the workers together, because the typed inline protocol is not compatible with older versions.
• Worker sizing: the max-workers setting defaults to 1, and max-gpu-workers should be at least the number of vLLM instances sharing the server.
• Isolated IPC mode works only with the vLLM MP connector, needs the cuda-python package on both sides and is incompatible with expandable_segments:True and vLLM sleep mode.
• Scope: sharing is per node. Peer-to-peer sharing and prefill-decode disaggregation for cluster-wide reuse are listed as planned, with no dates.

There is also a capacity question the post does not answer. One pool means eviction pressure from every rank lands on the same memory, and the docs say eviction triggers at a 0.8 watermark and frees 0.2 of allocated memory per cycle by default. If your working set is larger than L1, a shared pool can evict one rank's useful prefix to make room for another's, which is a behaviour to watch, not one the benchmark exercised.

Running it: ports, processes and a Kubernetes shape

The deployment page uses four ports. The transport port is the one the engine connector must agree with, so a mismatch between the server port and the lmcache.mp.port value in the connector config is the first thing to check when engines cannot find the cache.

# Server as the deployment page runs it (Docker, one node, 60 GB L1).
# --ipc host is what lets the engine and server share GPU memory handles;
# --isolated-ipc plus lmcache.mp.isolated_ipc on both sides removes that need.
docker run --runtime nvidia --gpus all --network host --ipc host \
  lmcache/standalone:nightly \
  /opt/venv/bin/lmcache server --l1-size-gb 60 --eviction-policy LRU \
  --max-workers 4 --port 6555

# vLLM side: the connector must point at the same transport port.
--kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":6555}}'

# Kubernetes: the server is a DaemonSet (one per GPU node), vLLM a Deployment.
kubectl create namespace multi-process
kubectl apply -f examples/multi_process/lmcache-daemonset.yaml
kubectl apply -f examples/multi_process/vllm-deployment.yaml

On Kubernetes the page deploys the server as a DaemonSet so each GPU node runs one, with host networking and the transport bound to 127.0.0.1. The DaemonSet does not request GPUs, the vLLM pods keep them, and LMCache pods scheduled onto nodes without GPUs crash with CUDA initialisation errors. The page lists at least 4 GPUs per node and the NVIDIA GPU Operator as requirements for its example.

What to do on Monday

Do not start by installing it. Start by finding out whether fragmentation is your problem at all, then run a controlled comparison.

1. Confirm the shape: you are running more than one DP rank and your traffic is multi-turn, agentic or RAG. With one rank there is nothing to share.
2. Record a baseline on your own workload: mean and p99 time-to-first-token from lmcache bench engine or your own replay, with your current offload flags.
3. Start one LMCache server per node with an L1 size taken from the host memory left after the OS and vLLM, then switch only the connector and offload flags.
4. Set max-gpu-workers to at least the number of engines on the node, and scrape port 9090 so you can see eviction and hit behaviour rather than infer it.
5. Pin versions together, rerun at more than one QPS with repetitions, and only then decide; the published result is one point.

If your routing layer already sends a conversation to the same rank every time, the shared pool buys you less than the post suggests, because the fragmentation it removes is the cost of misrouting. In that case fixing routing may be the cheaper first experiment.

The lesson I take from this post is that in DP plus EP serving, KV cache locality is a property of your deployment topology, not just of your cache policy. Moving the cache out of the rank and into a shared node-level service removes a real source of recomputation, and the 0.29 s mean is plausible for that reason, but the 10x headline is a p99 figure from one dev-build run. Measure your own workload before you believe either number.