SGLang Unified Radix Cache: Prefix Caching for Hybrid Models

Prefix caching looks like a solved problem until the model stops being plain full attention. A sliding-window layer only needs the last few tokens before the reuse point, a recurrent layer needs a snapshot of its state at exactly that point, and the cache that served your Llama-style model last year has no way to express either.

On 2026-09-23 the SGLang team published a deep dive on how its Unified Radix Cache handles that, adapted from an LMSYS article dated 2026-08-11. I did not run these benchmarks. I read the SGLang post, the HiCache design documentation, the two pull requests it leans on and the original SGLang paper, re-fetched each one, and kept only figures I could see on the page. The one thing I did run is a forty-line toy of the reuse-boundary rule, because that rule is the part worth understanding.

Why a hybrid model needs a different prefix cache

A radix cache stores the KV of past prompts in a tree so a new request can skip prefill for any prefix it shares. For full attention that is simple: every token attends to every earlier token, so reuse needs the KV along the entire matched prefix. Hybrid models break the symmetry, because each component of the model states a different requirement for the same reuse boundary.

The old design answered this with one specialised cache class per combination and per capability, so every new architecture multiplied the code. The new design is one tree with pluggable components, which means the number of classes scales with the number of components instead of the number of combinations. SGLang splits the work into UnifiedTreeCore, which owns matching, splitting, inserting, locking and evicting, and UnifiedRadixCache, which orchestrates the memory pools.

One tree, one slot per component, and tombstones

Each node in the tree stores a span of tokens plus one slot per component. FULL and SWA slots hold KV page indices; a MAMBA slot holds a checkpoint at the end of the node's prefix. Every component handles its own splits, inserts, locks and evictions, which is what lets one component give memory back without disturbing the others.

The consequence is the tombstone. SWA data outside the window that reuse needs can either stay cached or be evicted on its own, and an evicted SWA slot is left empty on a node whose FULL data is still there. To reclaim only part of a node's SWA data, the node is first split at the retention boundary. A MAMBA checkpoint is copy-on-write: it stays at its original endpoint, and a request that reuses it gets a private copy that it then mutates.

I find the tombstone the most useful idea in the post. A single-class cache has to treat a node as present or absent. Here a node can be half there, and that is exactly the state a sliding-window model spends most of its life in, because the bulk of a long prefix is never needed again for the window layers.

Finding the safe reuse boundary

Matching walks the path of matched nodes, and at each candidate boundary every component checks its own data independently. Compute resumes from the deepest boundary that all components accept. SGLang's example has node n3 with tombstoned SWA slots and node n4 without a MAMBA checkpoint, so reuse falls back to an earlier boundary.

The toy below is my own illustration of that rule, not SGLang code. Each node spans four tokens, the window is four tokens (the article uses a window of four in its diagrams, as an illustration rather than a benchmark setting), and I ran it with Python 3 to get the three printed results.

from dataclasses import dataclass

@dataclass
class Node:
    tokens: int            # length of the token span on this node
    full: bool = True      # FULL slot present (KV pages)
    swa: bool = True       # SWA slot present (False = tombstone)
    mamba: bool = False    # recurrent-state checkpoint at this node's end

def safe_boundary(path, window, use_swa=True, use_mamba=True):
    """Deepest token count from which compute can resume, or 0."""
    best, pos = 0, 0
    for i, node in enumerate(path):
        pos += node.tokens
        # FULL must be intact along the whole matched prefix.
        if not all(n.full for n in path[: i + 1]):
            break
        ok = True
        if use_swa:
            # SWA needs a continuous window of tokens ending at this boundary.
            need, j = window, i
            while need > 0 and j >= 0:
                if not path[j].swa:
                    ok = False
                    break
                need -= path[j].tokens
                j -= 1
        # MAMBA needs a checkpoint at exactly this boundary.
        if use_mamba and not node.mamba:
            ok = False
        if ok:
            best = pos
    return best

path = [
    Node(4, mamba=True),
    Node(4, mamba=False),
    Node(4, swa=False, mamba=True),   # tombstoned SWA slot
    Node(4, mamba=False),             # no checkpoint at this endpoint
]
print(safe_boundary(path, window=4))                                 # 4
print(safe_boundary(path, window=4, use_mamba=False))                # 16
print(safe_boundary(path, window=4, use_swa=False, use_mamba=False)) # 16

Read the three outputs together. With SWA alone the tombstone on n3 costs nothing, because the window at the end of n4 only needs n4. Add the recurrent component and the answer collapses from 16 tokens to 4, because the only checkpoint-bearing boundaries left are blocked by the tombstone and the missing checkpoint. That is the real cost of a hybrid model: reuse is limited by the strictest component at each point, not the average one.

HiCache: three tiers behind the same tree

HiCache extends the Unified Radix Cache across L1 (GPU memory), L2 (host memory) and L3 (external storage, Mooncake in SGLang's benchmarks). The design documentation is blunt about sharing, and it changes how you size a fleet.

A request first matches against L1 and L2 in the HiRadixTree. If the L3 hit length passes a threshold (256 tokens by default), the cache prefetches from L3 into host memory, then loads KV to the GPU while overlapping layer N+1 with the compute of layer N. After prefill, only data not already in L3 is written down. Three write policies decide when: write_through writes to the next tier immediately, write_through_selective only after an access-count threshold so only hot data is backed up, and write_back only on eviction from the upper tier.

For DeepSeek-V4, SGLang describes the extra pools (compressed KV, indexer buffers, compressor states) as sidecars that share their component's indices and transfers. Three sidecars follow FULL and two follow SWA, so a tier transfer moves the whole family together.

The launch command below uses only flags and environment variable names that appear in the HiCache documentation and the LMSYS HiCache post. The values are placeholders you must replace, and I have not run it on a GPU.

# Mooncake connection settings are read from environment variables.
# Names are from SGLang's HiCache post; every value here is yours to fill in.
export MOONCAKE_TE_META_DATA_SERVER="<metadata-server-url>"
export MOONCAKE_MASTER="<master-host:port>"
export MOONCAKE_PROTOCOL="<rdma-or-tcp>"
export MOONCAKE_DEVICE="<nic-names>"
export MOONCAKE_GLOBAL_SEGMENT_SIZE="<bytes-this-node-contributes>"

# L1 = GPU pool, L2 = host pool sized as a multiple of L1, L3 = Mooncake.
python3 -m sglang.launch_server \
  --model-path "$MODEL_PATH" --tp 8 \
  --page-size 64 \
  --enable-hierarchical-cache \
  --hicache-ratio 2 \
  --hicache-write-policy write_through_selective \
  --hicache-storage-backend mooncake \
  --hicache-storage-prefetch-policy timeout \
  --hicache-storage-backend-extra-config '{"prefetch_timeout_base": 2, "prefetch_timeout_per_ki_token": 0.1, "prefetch_timeout_max": 30}'

# --hicache-ratio must be greater than 1 (host pool relative to device pool);
# --hicache-size overrides it with an absolute GB-per-rank figure.
# write_through_selective backs up only entries past an access-count threshold.

What the numbers actually show

SGLang benchmarks multi-turn serving where each client keeps extending one conversation, so the prefix grows every round. The metric is effective input throughput, and hit rate is cached prefix tokens divided by complete prompt tokens, summed per round.

The headline is 15.5x for DeepSeek-V4-Flash with L3 against L1 alone, with a hit rate near 98 percent in later rounds and average TTFT under 9 s. For Inkling-Small the L3 run ends at a 96.8 percent hit rate and 1.23 s average TTFT; my own division of 67.1K by 15.5K gives about 4.3x. In both runs the story is the cliff: L1 and L1+L2 hit rates fall to roughly zero after round 35 to 40 on the first workload and after round 18 to 19 on the second, because the working set of live conversations no longer fits. The article describes those curves as approximate readings from its figures.

Treat the multipliers as a statement about this workload shape. Forty-eight to sixty-four clients each holding a long-lived conversation is chosen to overflow GPU and host memory, and the baseline is a cache that has already lost the prefix. It tells you L3 is decisive when the working set is larger than L1 plus L2, not that you will see 15.5x on your traffic.

Session-aware eviction and the Rust tree core

Two refinements sit on top of the tree. Session-aware eviction lets an application send a stable session_id so SGLang tracks which cached entries each session references; calling /close_session drops that session's references, while shared prefixes keep references from other sessions. For FULL eviction, unlocked eligible entries are ranked by whether they have session references, then by reference count, then by the base policy, usually LRU. Retention applies to L1 and L2; the storage backend manages L3 eviction.

Against HiRadixCache with LRU on SWE-bench, SGLang reports lower TTFT: on DeepSeek-V4-Pro at TP8, 11 percent at batch 128 (device hits up from 42 to 51 percent) but only 2.9 percent at batch 256; on Qwen3.5-397B-A17B at TP8, 13.5 percent at batch 32 (device hits from 5 to 34 percent) and 16.6 percent at batch 64 (device plus host hits from 58 to 67 percent). The gain shrinks as the batch grows, which is the honest shape of an eviction optimisation.

The second refinement moves the tree itself into Rust. Rust owns the radix topology, per-component lock accounting, intrusive LRU lists and eviction walks; Python keeps request-to-token mappings, physical KV allocation and pool orchestration. The merged pull request, sgl-project/sglang#32710, was merged on August 31, 2026 and reports 2,436 shared-suite cases passing on each backend. SGLang's L1-only prototype benchmark (200 turns, 100 input and 100 output tokens per turn, 6 trials) gives these TTFT reductions against the Python tree.

The ordering is worth noticing: the more complicated the component bookkeeping, the more Python overhead there was to remove. SWA gains most and plain full attention least.

Trade-offs and where this breaks

L2 is not a pool. Host memory is private to one instance, so adding replicas multiplies L2 capacity without letting any replica reuse another's cache. Only L3 can be shared, and only if the backend is set up for it. The file backend defaults to a node-local /tmp/hicache directory unless the path is a shared mount, which is an easy way to believe you have a shared cache when you do not.

The two refinements do not combine yet. Session-aware caching still runs on the Python tree core, and the Rust pull request lists the session radix cache among the deferred pieces, alongside the external cache linker, C128 and production use on AMD, MLX or NPU; those combinations fail early instead of running with incomplete semantics. The Rust speedup numbers come from an L1-only prototype pull request, which at the time of my reading was a draft marked work in progress, so the 38 percent figure is SGLang's own claim and not a number I could reproduce.

Also note what the SGLang post does not give you: no server flags, no launch command and no limitations section. Everything operational above came from the HiCache documentation, which does not mention hybrid, Mamba or SWA models at all. I could not confirm from a source I read that every HiCache option behaves identically for every component, so test your model before you trust a tier.

What to do on Monday

Start by finding out whether prefix reuse is your bottleneck, then add tiers one at a time. This order keeps each change attributable.

1. Measure your own hit rate with the definition SGLang uses, cached prefix tokens divided by complete prompt tokens, on real traffic over several hours, not a synthetic burst.
2. Check whether your model is hybrid. If it pairs FULL with SWA or MAMBA, the unified tree is the cache you want; read your version's release notes before assuming it is the default.
3. Enable L2 first with --enable-hierarchical-cache and --hicache-ratio, and compare against L1 alone. Expect a modest gain: SGLang's L2 step was 1.5x on one workload and 1.4x on the other.
4. Add an L3 backend only if your working set exceeds L1 plus L2, and make sure it is genuinely shared across instances. Choose the timeout prefetch policy if you have a TTFT SLO.
5. If you run multi-turn agents, send a stable session_id and call /close_session when a conversation ends, then watch device hit rate at your real batch size, since the gain shrank at larger batches in SGLang's SWE-bench runs.

The snippet below computes hit rate the way SGLang defines it, summed per round, from per-request counts you log yourself. It is my own helper, and it is the first thing I would run, because it needs no change to the server.

from collections import defaultdict

def hit_rate_by_round(requests):
    """requests: iterable of (round_no, cached_prefix_tokens, prompt_tokens).

    Same definition as the SGLang post: cached prefix tokens divided by
    complete prompt tokens, summed per round. Averaging per-request ratios
    instead would overweight short prompts.
    """
    cached, total = defaultdict(int), defaultdict(int)
    for round_no, cached_tokens, prompt_tokens in requests:
        cached[round_no] += cached_tokens
        total[round_no] += prompt_tokens
    return {r: cached[r] / total[r] for r in sorted(total)}

# Illustrative input, not measured data: a conversation that stops fitting.
sample = [(1, 0, 4096), (2, 4096, 8192), (3, 0, 12288)]
print(hit_rate_by_round(sample))  # {1: 0.0, 2: 0.5, 3: 0.0}

The rule I take from this: a prefix cache for a hybrid model is limited by its strictest component at each boundary, so build the cache around components that can fail independently and only then add tiers. Tombstones and checkpoints make reuse honest, and L3 makes it large, but the numbers SGLang reports belong to workloads built to overflow memory, so measure yours before you buy storage.