[faresobeid]

The P/D SLURM templates pin every decode node to one NIC with UCX_NET_DEVICES=mlx5_0:1. This PR removes the pin. Decode then uses the same list as prefill: every ACTIVE InfiniBand port, falling back to RoCE.

On a node where mlx5_0 is not an active InfiniBand port, decode cannot create its NIXL backend at all. On our H200 nodes, mlx5_0 is one of the 8 InfiniBand NICs (see swe-llmd.toml). So all 8 decode GPUs pull KV blocks and NIXL weights through that one NIC.

Numbers

Qwen3-30B-A3B with P/D on 2 H200 nodes, split into four 4-GPU groups: 2 trainer (EP8), 1 prefill, 1 decode. NIXL weight broadcast with overlap_transfer_and_replay, 16 steps per arm. Medians, first update dropped:

decode UCX_NET_DEVICES	in-pause weight transfer	p90	trainer time/broadcast_weights	mismatch-KL (mean)	prefill/decode weight checksums
mlx5_0:1 (main)	1.47 s	1.48 s	33.5 s	0.005	16/16 match
all active InfiniBand ports (this PR)	0.56 s	0.56 s	32.6 s	0.002	17/17 match

Neither arm logged a NIXL error, a failed request or a traceback. Decode created its NIXL KV connector and NIXL weight agent with the port list.

Prefill and decode shared a physical node in that run, so the KV transfer did not cross the network.

Prefill and decode on different nodes

Same setup, but prefill on node 0 and decode on node 1, so every KV transfer crosses InfiniBand. The trainer spans both nodes. Same tree for both arms (main + #4059); the only difference is the decode pin. 16 steps per arm. KV numbers are decode-side vllm:nixl_* totals from /metrics, summed over the 4 decode engines:

decode UCX_NET_DEVICES	KV transfers	KV bytes	mean KV transfer time	KV throughput	KV transfers <= 5 ms	failed transfers / notifications / expired	in-pause weight transfer (median, p90)	trainer time/broadcast_weights	time/step	mismatch-KL (mean)	weight checksums
mlx5_0:1 (main)	9597	89.5 GB	28.2 ms	0.33 GB/s	48%	0 / 0 / 0	2.52 s, 2.52 s	9.5 s	117.2 s	0.0010	17/17 match
all active InfiniBand ports (this PR)	9559	91.0 GB	16.2 ms	0.59 GB/s	70%	0 / 0 / 0	0.55 s, 0.55 s	7.2 s	115.9 s	0.0010	17/17 match

All prompt tokens on decode came from external_kv_transfer in both arms. Cross-node KV transfer works without the pin. It is 1.7x faster per transfer, and the weight transfer is 4.6x faster. Step time is about the same because KV and weight transfer are a small part of the step.

Checks

• rl @ examples/advanced/glm-5.3/swe.toml --dry-run and inference @ examples/advanced/glm-5.3/infer/pd-llmd.toml --dry-run render; bash -n passes on both scripts.
• pytest tests/unit -m "not gpu" -k "slurm or sbatch or template or disagg": 3 passed.

---

Note

Medium Risk
Changes cluster networking for NIXL/KV and weight transfer on decode; behavior depends on which RDMA ports are ACTIVE, though benchmarks in the PR show improved throughput without NIXL failures.

Overview
Removes the decode-only export UCX_NET_DEVICES=mlx5_0:1 from the P/D branches in inference.sbatch.j2 and multi_node_rl.sbatch.j2.

Decode nodes now keep the same UCX device list as prefill: active InfiniBand ports from sysfs (RoCE fallback, then all), which the templates already set before the role split. decode_env_vars can still override UCX per deployment if needed.

^{Reviewed by Cursor Bugbot for commit 74071cd. Bugbot is set up for automated code reviews on this repo. Configure here.}