Disaggregated Prefill-Decode in vLLM: NixlConnector Guide

Imagine a grocery store routing 64-pallet freight deliveries and ten-item express shoppers onto the exact same checkout belt: every time a truck arrives, forty-eight shoppers mid-checkout freeze for a full second. Running Large Language Model (LLM) prompt ingestion (prefill) and token streaming (decode) on one Graphics Processing Unit (GPU) node causes that exact stall. Disaggregated Prefill-Decode via NVIDIA Inference Xfer Library (NIXL) and LMCache eliminates it, cutting p99 Inter-Token Latency (ITL) by 74% and boosting GPU goodput 2.4x.

Why Colocated Continuous Batching Breaks Under Long-Context RAG

Continuous batching engines like vLLM and SGLang transformed LLM serving by interleaving incoming prompt requests with active token-generation streams inside unified PagedAttention memory. That colocated design works cleanly when prompts stay short (512 to 2,048 tokens), and you don't notice contention at low concurrency. Once enterprise Retrieval-Augmented Generation (RAG) and coding agents push input contexts to 32,000 or 64,000 tokens, colocated scheduling can't hide the underlying hardware physics mismatch.

LLM inference consists of two fundamentally opposite hardware phases:

• The Prefill Phase (Compute-Bound): Processes all 64,000 input prompt tokens in parallel to populate the initial Key-Value (KV) cache. A single long-context prefill saturates 100% of GPU Streaming Multiprocessors (SMs) and Tensor Cores for 700 ms to 1,200 ms, leaving High-Bandwidth Memory (HBM) bandwidth underutilized.

• The Decode Phase (Memory-Bandwidth-Bound): Generates output tokens one step at a time, reading gigabytes of cached KV state from HBM3e on every forward pass while utilizing less than 12% of Tensor Core compute.

When a colocated vLLM scheduler admits a 64K RAG prompt alongside 48 active streaming users, the compute-heavy prefill monopolizes the GPU's SMs. Every streaming user experiences a sudden 900+ millisecond UI freeze mid-sentence. Even when engineers enable Chunked Prefill (--enable-chunked-prefill), splitting a 64K prompt into 2,048-token slices still forces repeated KV-cache HBM re-reads and inflates p99 Inter-Token Latency (ITL) past strict 50 ms enterprise Service Level Agreements (SLAs).

How Disaggregated Serving & NVIDIA NIXL Split the Pipeline

Prefill-Decode Disaggregation resolves this contention by splitting a homogeneous GPU cluster into two physically isolated worker pools: a Prefill Pool (kv_producer) tuned for raw Floating-Point Operations Per Second (FLOPs) and a Decode Pool (kv_consumer) tuned for maximum HBM3e memory capacity and bandwidth.

Historically, disaggregated serving stalled on network serialization: copying a 4.5 GB KV cache tensor over standard TCP/IP sockets added seconds of Time-to-First-Token (TTFT) overhead. vLLM V1 eliminates that bottleneck by integrating NixlConnector, built atop NVIDIA's NIXL (NVIDIA Inference Xfer Library) and GPUDirect Remote Direct Memory Access (RDMA) over 400 Gbps RoCEv2 (RDMA over Converged Ethernet) or InfiniBand.

Instead of bouncing tensors through host Central Processing Unit (CPU) DRAM or Python pickle buffers, NixlConnector registers PagedAttention HBM blocks directly with the network interface card (NIC) via Unified Communication X (UCX):

1. Prefill Execution & LMCache Check: The router sends the incoming 64K prompt to a Prefill worker. LMCache first checks cluster-wide shared DRAM/NVMe tiers for cached prefix blocks (see our Context Prefix Caching Architecture) and computes only the uncached suffix tokens.

2. Asynchronous Layer-Pipelined RDMA Transfer: As each Transformer layer finishes computing its Key and Value blocks on the Prefill GPU, NixlConnector streams those blocks asynchronously across 400 Gbps RDMA (50 GB/s per NIC) directly into pre-allocated PagedAttention slots on the target Decode GPU.

3. Zero-Stall Autoregressive Decoding: Once the final layer's KV block lands in Decode HBM and the first token emits, the Prefill GPU immediately frees its memory buffer. The Decode GPU steps through autoregressive generation at a flat 18.4 ms p99 ITL because it doesn't get interrupted by incoming prompts.

Production Benchmark: Colocated vs. Chunked vs. Disaggregated NIXL

We benchmarked an 8-node (64x NVIDIA H200 SXM, 400 Gbps RoCEv2) sovereign cluster serving a 70B-parameter dense model alongside a 671B Mixture-of-Experts (MoE) model under mixed RAG traffic (28,000 mean input tokens, 650 output tokens, strict SLA: TTFT < 2.5s and p99 ITL < 45ms):

Notice why goodput jumps 2.4x on the exact same 64 GPUs. Under colocated serving, operators have to artificially throttle concurrency (--max-num-seqs 16) just to stop prefill spikes from breaking the 45 ms streaming SLA—leaving half their expensive H200 silicon idle. Disaggregating the pools lets you run Prefill nodes at TP=4 (compute-saturated) and Decode nodes at TP=8 or Expert Parallelism with massive batch sizes (batch_size = 128+), compounding the savings from Multi-Head Latent Attention (MLA) 93% KV Compression.

Configuring vLLM V1 Disaggregated Pools with NixlConnector

Deploying disaggregated serving requires explicit UCX GPUDirect transport flags and matching --kv-transfer-config declarations across your Kubernetes GPU pods. Below is our hardened launch configuration for a 3 Prefill : 5 Decode node ratio over RoCEv2:

# 1. Required GPUDirect RDMA & NIXL Transport Environment (Both Pools)
export VLLM_USE_V1=1
export UCX_TLS="rc,sm,cuda_copy,cuda_ipc"
export UCX_NET_DEVICES="mlx5_0:1,mlx5_1:1"
export CUDA_DEVICE_MAX_CONNECTIONS=1

# 2. Launch Compute-Optimized PREFILL Node (kv_producer, TP=4, FP8 KV Cache)
vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --port 8100 \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --kv-transfer-config '{
    "kv_connector": "NixlConnector",
    "kv_role": "kv_producer",
    "kv_rank": 0,
    "kv_parallel_size": 2,
    "kv_buffer_size": 8589934592
  }'

# 3. Launch Bandwidth-Optimized DECODE Node (kv_consumer, TP=4, High Concurrency)
vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --port 8200 \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-seqs 192 \
  --gpu-memory-utilization 0.94 \
  --kv-transfer-config '{
    "kv_connector": "NixlConnector",
    "kv_role": "kv_consumer",
    "kv_rank": 1,
    "kv_parallel_size": 2,
    "kv_buffer_size": 17179869184
  }'

Five Engineering Rules for Disaggregated GPU Clusters

1. Verify nvidia-peermem Before Enabling NixlConnector: If the nvidia_peermem kernel module isn't loaded on your bare-metal Kubernetes nodes (lsmod | grep nvidia_peermem), UCX silently falls back to staging KV blocks through PCIe host RAM, cutting transfer bandwidth from 50 GB/s to 11 GB/s and doubling TTFT.

2. Quantize Transferred KV Tensors to FP8 (fp8_e4m3): Halving KV precision from BF16 to FP8 cuts cross-node RDMA wire bytes by 50% before a single packet touches the top-of-rack switch, preventing network congestion during burst RAG traffic.

3. Right-Size Your Prefill-to-Decode (P:D) GPU Ratio by ISL/OSL: Measure your workload's Input Sequence Length (ISL) to Output Sequence Length (OSL) ratio. RAG summarization (32K ISL : 400 OSL) typically needs a 3:2 Prefill-to-Decode GPU ratio, whereas long chain-of-thought reasoning (4K ISL : 8K OSL) runs best at 1:3.

4. Disable Speculative Decoding on Saturated Decode Pools: As shown in our vLLM Speculative Decoding High-Batch Wall analysis, disaggregated Decode nodes run at high batch sizes (64–192 sequences) where draft-model verification wastes Tensor Core cycles; reserve speculative decoding strictly for low-batch latency tiers.

5. Isolate RDMA Control Sockets Behind Cluster NetworkPolicies: Never expose NIXL/UCX sideband coordination ports outside the dedicated GPU pod subnet, and model your hardware savings in our Sovereign MoE Silicon X-Ray & Cloud GPU Repatriation Calculator before ordering additional racks.

When Should You Use Disaggregated Prefill-Decode Instead of Chunked Prefill?

Chunked prefill works well on single-node deployments (1–8 GPUs) with moderate input lengths (< 8,192 tokens) and relaxed Inter-Token Latency budgets (p99 ITL > 80 ms). You should switch to disaggregated prefill-decode via NixlConnector once you operate multi-node clusters serving long-context workloads (16K–128K tokens) under strict streaming SLAs (p99 ITL < 40 ms), where colocated prefill interference wastes over 40% of cluster goodput.

Does Disaggregated Serving Require InfiniBand, or Does RoCEv2 Work?

Both InfiniBand and RoCEv2 (RDMA over Converged Ethernet) are supported natively by NVIDIA NIXL via its UCX transport backend. On 200 Gbps or 400 Gbps RoCEv2 fabrics configured with Priority Flow Control (PFC) and Explicit Congestion Notification (ECN), GPUDirect RDMA transfers a 2 GB FP8 KV cache between nodes in under 42 milliseconds with zero packet loss.

How Do LMCache and NixlConnector Work Together in vLLM V1?

NixlConnector provides the low-level asynchronous HBM-to-HBM and storage-to-HBM data-movement engine, while LMCache acts as the distributed KV cache manager across shared system prompts and multi-turn agent histories. Pairing LMCache with NIXL lets Prefill nodes pull warm RAG document KV blocks from NVMe or pooled host DRAM at wire speed instead of recomputing attention from scratch.