[faresobeid]

Router replay now works with disaggregated P/D on vLLM 0.31, together with sampling replay. It replaces main's config rejection of router replay + P/D.

Problem on 0.31. vLLM 0.31 captures routed experts through the AuxOutput connector (V2 runner only, vllm#45635) and rejects P/D connectors (_verify_aux_output_compatibility). The rejection is real: a decode (D) instance never computes a remotely prefilled prompt, so its worker would try to materialize the prompt rows from its block store (BlockObjectStoreError), and D's first forward (token prompt_len-1) starts mid-block, which the routed-experts buffer rejects.

Fix (plugin patch monkey_patch_routed_experts_with_pd_connectors, our upstream AuxOutput patch 02):

• Config: skip only the P/D connector check when the KV transfer config has NixlConnector (alone or inside MultiConnector). Every other AuxOutput check still runs.
• Scheduler: a request on D (kv_transfer_params with a remote_engine_id and no do_remote_decode; NIXL clears do_remote_prefill but keeps that) emits rows only from its first forward on D (num_computed_tokens at that point: normally prompt_len-1, lower after a failed KV load, where D recomputes those rows itself). The floor survives preemption and is dropped when the request finishes.
• Worker: the partially known first block is not buffered or published; later blocks D computes in full are published as usual.

Router v0.2.3 required (router#61); the decode start comes from #3900 (merged). On a decode instance the rows start at its first forward, so the response carries only the rows D computed. #3900's packed /generate anchors the rows at the end: start = len(prompt) + len(completion) - 1 - len(rows) (equal to routed_experts_prompt_start without P/D). This PR does not touch routed_experts.py/serving_tokens.py, so it and #3900 (and #3920/#3794 on it) land in either order; P/D router replay needs both. Router v0.2.3 cuts decode rows by position ((prefill.start + prefill.seq_len) - decode.start) and also handles uint16 routing (models with more than 256 experts). This PR pins vllm-router to v0.2.3 in pyproject.toml/uv.lock. Without #3900, the merge would drop real completion rows.

Also: removes validate_disaggregated_no_routed_experts (both validators on main), docs.

Stack: this PR → #3831 (router replay with KV offload) → #3832 (P/D per-role KV offload) → #4023 (draft, HiSparse on decode).

Checks run

• pytest -m "not gpu" tests/unit/test_configs.py tests/unit/inference on this head: 214 passed. ruff check / ruff format --check: clean.
• New tests (test_routed_experts_pd.py): the scheduler floor (first forward, preemption, cleanup) on vLLM's real AuxOutputSchedulerConnector, and the worker skip on the real RoutedExpertsBuffer. test_combined_replay_uses_v2_runner is parametrized over single-node and P/D.
• vLLM's own tests/distributed/aux_output_connector/test_store.py (v0.31.0) with this patch applied plus patch 02's two new tests: 66 passed. Without the patch the two new tests fail.

GPU validation (2026-10-06, vLLM 0.31.0, 1x 8xH200, Qwen3-30B-A3B TP1, VLLM_BATCH_INVARIANT=1, router v0.2.2 with zero-padded decode rows)

• 1P1D (NIXL) vs colocated, router replay + sampling masks: 10/10 identical at concurrency 1 (prompts 64-6000 tokens, 128 completion tokens) and 20/20 at concurrency 10 with a fresh cache_salt. Identical means tokens, prompt and completion routing rows (100%) and sampling masks; rows = prompt + completion - 1.
• Forced decode preemption (D with 1300 blocks, 20 requests, 1000-4000 prompts, up to 1024 tokens, concurrency 10): 6 preemptions on D, 20/20 identical to colocated.
• All engines log Using V2 Model Runner; no BlockObjectStoreError.
  Re-run (job 2900, 2026-10-06, same setup, launched from per-role configs written by feat(inference): P/D per-role KV offload #3832's write_pd_configs): 1P1D (NIXL, no offload) vs colocated, router replay + masks: 10/10 identical at concurrency 1 and 20/20 at concurrency 10 with a fresh cache_salt; 0 errors.

Not run: injected NIXL load failure (kv_load_failure_policy=recompute), MTP on decode, an rl run with a P/D config.

GPU validation with router v0.2.3 (2026-10-14)

1x 8xH200, vLLM 0.31.0, Qwen3-30B-A3B TP1, VLLM_BATCH_INVARIANT=1, router replay + sampling masks. prime-rl = #3830 (978146c) merged with #3900 (20a3225); router = release/v0.2.3 (ff818da, this PR + version bump), wheel built locally. 1P1D (NIXL) through the router vs a colocated server. Identical means: same tokens, same routing rows for prompt and completion (100%, rows = prompt + completion - 1, merged start = 0), same sampling masks, same logprobs.

• Concurrency 1 (prompts 64-6000 tokens, 128 completion tokens): 10/10 identical.
• Concurrency 10, fresh cache_salt on the colocated side: 20/20 identical.
• uint16: no knob exists (vLLM picks uint8 for <=256 experts), so a test-only sitecustomize forced vLLM's capturer to uint16 on P, D and the colocated server. The responses carry dtype: uint16. Concurrency 1: 10/10 identical; concurrency 10: 20/20 identical.
• Forced decode preemption (D with num_gpu_blocks_override=1300, 1000-4000 token prompts, up to 1024 tokens, concurrency 10): 6 preemptions on D, 20/20 identical.
• Router logs: no errors or warnings during the runs (only the shutdown warning). Engines: all Using V2 Model Runner, no BlockObjectStoreError.
• Negative control, router v0.2.2 with the same P/D pair: 8/10 requests fail (HTTP 500 decode routed_experts has 128 rows, expected at least <prompt_len>). The 2 prompts of 64 tokens return 128 rows instead of 191, and the completion rows are wrong.

---

Note

High Risk
Changes expert-routing capture on NIXL P/D via vLLM monkey patches and removes prior safety guards; training correctness depends on router 0.2.3 merge behavior and patch edge cases (preemption, failed KV load).

Overview
Router replay and sampling replay are now allowed on disaggregated prefill/decode (P/D) deployments, where they were previously blocked by config validation and vLLM’s AuxOutput + NIXL incompatibility.

Config validators that rejected enable_return_routed_experts with deployment.type = "disaggregated" are removed from inference and RL configs. Docs now describe P/D behavior: prompt routing from prefill, completion routing from decode, and that sampling capture can run alongside router replay under P/D.

A startup vLLM monkey patch (monkey_patch_routed_experts_with_pd_connectors) bypasses only the P/D connector check in _verify_aux_output_compatibility, raises an emit floor on decode so routed-expert rows start at the instance’s first forward, and skips buffering a partial first block after remote prefill. vllm-router is pinned to v0.2.3 so the router can merge prefill/decode expert rows correctly.

Unit tests cover scheduler floors, worker buffering, and V2 runner setup for both single-node and disaggregated configs.

^{Reviewed by Cursor Bugbot for commit e586ef8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b1b2742. Configure here.}

[mikasenghaas]

lgtm, lets wait for @samsja given that he closed before