Sunita Nadampalli, a Principal Engineer at AWS, and Nicolò Lucchesi, a Research Engineer at Mistral AI and a vLLM maintainer, have published a contribution describing vLLM push mode KV transfer alongside benchmark data from Amazon EC2 p5en.48xlarge instances. At 8 queries per second with 16,000-token inputs, a profile representative of production coding assistants and long-document analysis pipelines, push mode reached approximately 540 milliseconds of time-to-first-token, against roughly 1,744 milliseconds for the prior pull-based connector. The accompanying gain in time-per-output-token narrows from approximately 10.4 ms to about 7 ms.
Modern high-throughput LLM serving has converged on prefill-decode disaggregation: a compute-heavy prefill phase that builds the key-value cache, and a memory-bandwidth-heavy decode phase that emits output tokens. The architectural price is the KV handoff between phases, which sits directly on the wall-clock path to the user’s first token. In vLLM’s pull model, the router sends prefill block IDs only after prefill completes, then the decode instance allocates memory, prepares NIXL descriptors, and issues a READ. Each step stacks latency sequentially after prefill, accumulating to nearly two seconds at scale.
vLLM push mode KV transfer: how the new protocol moves coordination off the critical path
The new connector inverts initiation. The decode instance pre-registers its target GPU memory while prefill is still running; when prefill finishes, the data is written directly into already-prepared destination memory, with no round-trip or inline descriptor setup. A registration record describing exact GPU memory locations is transmitted to the prefill instance as an NIXL notification at allocation time, so the destination is already known when the data is ready to move.
Rendezvous handling accepts either arrival order: if registration arrives first it waits for KV completion, and if KV finishes first it waits for registration. Once paired by request ID, a block-lease-and-watchdog mechanism resolves preempted or retried requests. The prefill instance then issues an NIXL WRITE that copies KV blocks directly into the decoder’s pre-registered GPU memory, targeting each tensor-parallel rank. This work runs on a dedicated nixl-push-writer background thread per tensor-parallel worker, never touching the prefill engine’s main forward pass.
That background thread is the architectural key, not an implementation detail. Every expensive per-request operation happens off the prefill loop: establishing connections, mapping block IDs to NIXL memory descriptors, merging contiguous blocks into larger transfer regions, and submitting the WRITE. At high concurrency, when dozens of requests complete prefill simultaneously, those transfers run in parallel while prefill compute continues uninterrupted processing the next batch. The P5en interconnect, EFAv3 at 3,200 Gbps with GPUDirect RDMA, is fast enough that bandwidth is not the constraint. The bottleneck is the serial coordination protocol, and the push connector removes that overhead in software rather than requiring faster hardware.
By the numbers: 540 ms versus 1,744 ms at scale
Adoption will depend on how quickly operators can validate the change against their own traffic, and on whether upstream documentation keeps pace with the deployment guidance published alongside the change.
Adoption will depend on how quickly operators can validate the change against their own traffic, and on whether upstream documentation keeps pace with the deployment guidance published alongside the change.
Adoption will depend on how quickly operators can validate the change against their own traffic, and on whether upstream documentation keeps pace with the deployment guidance published alongside the change.
Adoption will depend on how quickly operators can validate the change against their own traffic, and on whether upstream documentation keeps pace with the deployment guidance published alongside the change.
The benchmark ran a 1P1D topology, one prefill node and one decode node with Qwen3-32B, on Amazon EC2 p5en.48xlarge instances connected via EFA with prefix caching disabled to force full KV transfers. The software stack was vLLM 0.29.0, NIXL 1.4.1, and CUDA 13.0. Configuration swaps NixlPushConnector for NixlConnector on both prefill and decode servers.
The 540-millisecond figure sits near the 500 ms “feels responsive” threshold flagged in LLM latency user-experience research, while pull mode’s 1,744 ms lands in the “breaks conversational flow” zone. At low load with short prompts, both modes perform near-identically, around 5.6 to 5.8 ms TPOT, confirming the overhead is specifically a concurrency-and-context-length problem that surfaces precisely where production workloads live.
Push mode is not universally superior. It must choose the decode instance before prefill finishes, which limits load-aware routing decisions. Pull mode remains preferable for bidirectional KV transfer in multi-turn agentic workflows and for dynamic load balancing. Operators should match the connector to the workload rather than expecting one setting to dominate.
Security teams should also pin vLLM to 0.29.0 or later. CVE-2026-22807, an arbitrary code execution flaw via dynamic module loading with CVSS 8.8, was disclosed January 21, 2026 and was patched in vLLM 0.14.0, so the current benchmark version is unaffected. Looking forward, the broader lesson from vLLM push mode KV transfer is that further time-to-first-token reductions in disaggregated inference will come from protocol design rather than from buying faster interconnects.
Source: https://www.techtimes.com/articles/329248/20261020/vllm-disaggregated-inference-gets-push-kv-transfer-slashing-first-token-latency-three-times.htm