[lucamotz]

Overview

NIXL can now transfer KV between P and D whose hybrid KV cache group plans differ, so NixlConnector no longer needs #58207's TP-invariant planning, except for MLA. Motivated by RFC #58638. Stacked on #58207 at e55909cad0, #58860, #59746, #59747 and #59330; only the last four commits are new.

Claims

• Connectors say whether their peers need matching KV cache groups: requires_matching_kv_cache_groups() defaults to True, so every other connector keeps [Core] Pick hybrid KV cache group size by worst-case padding bytes; add --min-kv-cache-group-layers #58207's planning.
• NixlConnector pull between different plans is exact: Qwen3.6-35B-A3B-FP8 + DFlash at P TP2 ↔ D TP4 gives bitwise-identical outputs to TP-invariant plans, with 7–28% more KV tokens on D.
• This also covers plans that differ under TP-invariant planning, which depends on each side's --max-num-batched-tokens, PP and async scheduling.

Validation

pytest tests/v1/core/test_kv_cache_utils.py
pytest tests/v1/kv_connector/unit/test_nixl_connector.py tests/v1/kv_connector/unit/test_nixl_connector_hma.py \
       tests/v1/kv_connector/unit/test_nixl_push_connector.py tests/v1/kv_connector/unit/test_nixl_desc_geometry.py \
       tests/v1/kv_connector/unit/test_tp_mapping.py tests/v1/kv_connector/unit/test_nixl_heartbeat.py \
       tests/v1/kv_connector/unit/test_bidirectional_kv_transfer.py tests/v1/kv_connector/unit/test_handshake_pp_aggregation.py \
       tests/v1/kv_connector/unit/test_nixl_simple_cpu_offload.py tests/v1/kv_connector/unit/test_transfer_topology_sharded.py

On 6× H100, pull with independent plans is bitwise equal to TP-invariant push in both directions and at 262144 tokens. Under load, it shows no TTFT, TPOT or throughput cost against TP-invariant pull beyond run-to-run spread. On 2× DGX Spark, smaller models in pull and push, including D prefix hits and an MTP drafter, are bitwise equal to matching plans too.

CPU tests and byte oracle

• test_pull_between_different_group_plans (new) builds P and D workers for sliding-window and GDN hybrids whose plans put 2 vs 4 layers in a group, runs the real register_kv_caches, add_remote_agent and pull read path, and replays the recorded NIXL descriptors on CPU tensors: every D head and token holds P's bytes for the same layer, head and token, and nothing else is written.
• test_kv_transfer_group_planning_is_tp_invariant gets one row per connector; test_layer_handshake_rejects_unsupported_geometry gets a row for a PP push producer whose plan differs.
• 77126176e8 on main 74c5cbcd7b: the 10 NIXL files pass with 709 tests and test_kv_cache_utils.py with 187, except 3 tests that also fail on main in this environment.
• A local byte oracle, not part of this PR, replays pull and push across P/D TP pairs up to 8, different plans, block sizes and D prefix hits: 2032 cases are byte-exact and none is silently wrong; the other 238 raise.

GPU: Qwen3.6-35B-A3B-FP8 + DFlash, 6× H100 SXM

07c43a220c on main b1f229fb75. Qwen/Qwen3.6-35B-A3B-FP8 + z-lab/Qwen3.6-35B-A3B-DFlash (7 speculative tokens), TRITON_ATTN, --enforce-eager, greedy, 4 code prompts of 1100–12000 tokens, P TP2 on GPUs 0–1 and D TP4 on 2–5 and the reverse, NCCL_NVLS_ENABLE=0 and VLLM_USE_DEEP_GEMM=0. Each NixlConnector pull with independent plans is compared with a NixlPushConnector run, which keeps TP-invariant plans, on the same image: texts and logprobs are bitwise equal on all 4 prompts, DFlash accepted tokens are equal, and D reads every prompt token but the last over NIXL.

P → D, max_model_len	groups P / D, independent vs TP-invariant	KV tokens vs TP-invariant
TP2 → TP4, 16384	10 / 8 vs 10 / 10	D +7.3%
TP4 → TP2, 16384	8 / 10 vs 10 / 10	P +7.3%
TP2 → TP4, 262144	17 / 5 vs 10 / 10	P +9.1%, D +27.7%

These runs set --block-size 592 on the TP4 engine, as the new log line suggests. Without it, TP4 picks block 304 and outputs stay bitwise equal, but D's add agent handshake step takes 11–15× longer and NIXL transfers about 6× longer.

Perf A/B: independent vs TP-invariant pull under load, 6× H100 SXM

Arm B is 07c43a220c on main b1f229fb75 (plans 17 / 5); arm A is the same build with "Let pull peers plan KV cache groups independently" reverted (TP-invariant, 10 / 10). P TP2 → D TP4, max_model_len 262144, --block-size 592 on TP4, the flags above. vllm bench serve through the P/D proxy with random 8192-token prompts, run order A B B A, each run restarting both engines; greedy outputs are bitwise equal across all four runs. B − A pools both runs per arm, with a 95% bootstrap CI in ms.

load	metric	A	B	B − A	95% CI
1 at a time, 80 requests, 64 output tokens	TTFT p50	573.4 ms	564.9 ms	−1.5%	[−16.1, +0.8]
	TTFT p99	623.6 ms	607.6 ms	−2.6%	[−42.7, +22.3]
	TPOT p50	30.38 ms	30.41 ms	+0.1%	[−9.5, +9.1]
16 at a time, 320 requests, 256 output tokens	TTFT p50	1016.0 ms	1013.2 ms	−0.3%	[−43.7, +16.8]
	TTFT p99	3989 ms	4038 ms	+1.2%	[−509, +578]
	TPOT p50	17.21 ms	17.05 ms	−0.9%	[−3.2, +3.0]
	output tok/s	523.4	526.3	+0.6%	run spread 3.1% / 1.1%

D's NIXL transfers move the same bytes in both arms; under load their mean time is 77.6 vs 78.9 ms, and post-processing takes about 0.4 ms longer per transfer in B. Eager mode, so absolute numbers are not production numbers.

GB10: smaller models, pull and push, 2× DGX Spark

Build 98132d0d55: this branch's four commits on main d0d6e5f3a2. P and D on one Spark, or the TP2 side across both Sparks; --enforce-eager, greedy. Each case differs between P and D in plan, block size or TP and is compared with the same setup on matching plans and pages, on the same image: texts and logprobs are bitwise equal, and D reads every prompt token but the last over NIXL.

• Qwen3.5-0.8B (GDN): plans of 3 vs 9 groups, pull and push at TP1 → TP1 and TP1 ↔ TP2, including D prefix hits; with its MTP layer as drafter at TP1 → TP1, accepted tokens are equal.
• Gemma-4-E2B: pull at TP1 → TP1 and TP2 → TP1, push at TP1 → TP2.
• Qwen3.6-35B-A3B-FP8 + DFlash: pull at TP1 ↔ TP2, accepted tokens equal.
• NixlConnector with OffloadingConnector through MultiConnector plans as without a connector and matches NIXL alone on prompts that share no prefix; with shared prefixes, P's CPU offload serves them and logprobs differ slightly.
• Cross-node transfers ran over RoCE: UCX rc_mlx5, with the reads into GPU memory emulated in software.

Details

• Hook: replaces the planner's kv_transfer_config is not None check. MultiConnector asks the connectors it wraps; OffloadingConnector with CPUOffloadingSpec and SimpleCPUOffloadConnector return False, since their KV doesn't leave the engine.
• Group pairs: the handshake, version 15, advertises each region member's KV cache group, KV heads and tokens per page. Where plans differ or the model-wide head mapping doesn't fit every layer, layers transfer per pair of local and remote KV cache groups, in pull and push. Matched plans that the model-wide mapping fits keep today's path.
• Who plans independently: NixlConnector returns use_mla. NixlPushConnector stays TP-invariant, since a PP push producer routes layers by name. MLA, CSA-linear and PP push producers reject peers with different plans at the handshake.
• Log: names the --block-size that makes P and D pages divide, once per remote engine.

Not covered: MLA, where a GQA draft under an MLA target stays with #58471 and #59071; DCP; GPUDirect RDMA across nodes. The NIXL compatibility doc isn't updated yet.

The four commits change +700/−51 production lines. The hook can go as a small PR on top of #58207 and the NIXL commits after it, if that is easier to review. No open NIXL PR, and none citing #58638 or #58207, transfers between different KV cache group plans.

AI assistance (Claude Code) was used for the analysis, patch and tests. I reviewed every changed line.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lucamotz.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork