perf: enable HybridEP in H100/B200 performance recipes

[seonjinn]

Dependencies

Depends on #3436 and #2964. Merge both prerequisites before this PR.

Nemotron3 Super additionally requires a Megatron-Bridge/Megatron-LM revision containing NVIDIA/Megatron-LM#5542 and NVIDIA/Megatron-LM#6114. This PR does not vendor a custom Megatron-LM fork.

Summary

• Enable HybridEP directly in the canonical 8-GPU MoE performance recipes for H100/B200.
• Set explicit x86 topology: 8 ranks per NVLink domain, domain size 8, MNNVL disabled, and combine chunk size 128.
• Set explicit GB200 topology for existing 4-GPU HybridEP recipes while leaving dense and non-HybridEP recipes unchanged.
• Enable one-time HybridEP prepadding only for supported PP=1, MTP-disabled packed recipes.
• Validate and report the effective HybridEP topology at rank zero.
• Bound Qwen3-30B-A3B H100 logprob memory with logprob_chunk_size: 1024 and defer_fp32_logits: true.

H100 memory fit and performance

On the current vLLM 0.25.1 dependency stack, the previous unchunked AllToAll Qwen3-30B-A3B 4n8g recipe reproduced a policy-training OOM at step 2. Chunking at 1024 still OOMed with AllToAll at step 12, while HybridEP completed 20/20 steps. The selected HybridEP + chunk-1024 setting also outperformed the memory-safe AllToAll + chunk-512 control.

The timings below are means over 17 matched steady-state samples from 20-step H100 80 GB runs.

Dispatcher / chunk	Result	E2E step (s)	E2E tokens/s/GPU	Policy step (s)	Logprob step (s)
AllToAll / 512	20/20	157.41	1,322.41	38.12	11.80
AllToAll / 1024	OOM at step 12	—	—	—	—
HybridEP / 512	20/20	156.54	1,330.36	37.78	10.75
HybridEP / 1024	20/20	154.91	1,343.06	36.94	10.66

Relative to the memory-safe AllToAll / 512 control, the selected setting reduces E2E step time by 1.59%, increases E2E throughput by 1.56%, reduces policy time by 3.12%, and reduces logprob time by 9.64%. The matched chunk-512 A/B showed no directional regression in reward, generation-KL, JS-divergence, probability-ratio, or masked-sequence metrics.

This establishes the recipe-level memory-fit regression on the vLLM 0.25.1 stack; it does not isolate vLLM as the sole causal component.

Testing

• Qwen3-30B-A3B H100 80 GB: matched AllToAll/HybridEP 20-step runs linked above.
• 59 resolved-recipe unit cases passed.
• B200 resolved-recipe validation passed.
• git diff --check

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[seonjinn]

/ok to test 67a4b3d

[seonjinn]

/ok to test 67a4b3d

[seonjinn]

/ok to test 0480dc6

[seonjinn]

Contributor self-review status: this recipe PR remains blocked on a clean HybridEP runtime dependency. The current moe_hybridep_prepad_packed_inputs setting has no typed runtime consumer on latest main, while the intended dependency path still relies on the over-broad #2964 branch. Recommendation: first land/replace the runtime support and #3436 pin, then restack and validate this recipe; do not retrigger CI on the current head.

[seonjinn]

/ok to test 1bef174