perf(recipes): use FlashInfer TRT-LLM MoE for BF16

[seonjinn]

What does this PR do?

Use the FlashInfer TRT-LLM MoE backend in the Blackwell BF16 performance
recipes where it produced a measured end-to-end gain:

• Qwen3-30B-A3B synchronous GRPO: keep rollout TP1/PP1 and replace Triton.
• Qwen3-235B-A22B: use rollout TP4/PP2 and replace Triton. TP4 satisfies the
  TRT-LLM BF16 expert-width requirement; PP2 preserves the rollout GPU count
  while avoiding the host-memory pressure observed with TP4/PP1.

The asynchronous Qwen, DeepSeek-V3, and Nemotron recipes remain unchanged
because the tested configurations did not establish an end-to-end win.

Native BF16 TRT-LLM layerwise refit support is already available on main
through #3545. The two-gather refit optimization in #3669 is a separate
follow-up and is not required by this recipe-only change.

Performance

The table reports FlashInfer TRT-LLM versus Triton over the common steady-state
window, Steps 3-9,11-19. Each row is a same-source, same-topology 20-step
backend pair on OCI-HSG GB200; only the MoE backend changes.

Workload	GPUs and rollout topology	E2E step time	Generation time	Generation throughput	Refit time
Qwen3-30B-A3B sync	16, TP1/PP1	182.124s -> 177.933s (-2.30%)	69.173s -> 63.594s (-8.06%)	+8.87%	1.895s -> 4.182s
Qwen3-235B-A22B	64, TP4/PP2	273.495s -> 248.103s (-9.28%)	192.641s -> 161.776s (-16.02%)	+19.18%	3.669s -> 8.395s
Qwen3-235B-A22B	128, TP4/PP2	210.739s -> 189.589s (-10.04%)	161.290s -> 135.871s (-15.76%)	+19.10%	3.913s -> 8.688s

FlashInfer's generation savings exceed its higher BF16 refit cost in these
three pairs. These are matched single-job pairs, not repeat-level variance
estimates.

W&B: Qwen3-30B FlashInfer,
Qwen3-30B Triton,
Qwen3-235B 64-GPU FlashInfer,
Qwen3-235B 64-GPU Triton,
Qwen3-235B 128-GPU FlashInfer,
Qwen3-235B 128-GPU Triton.

Why DeepSeek-V3 remains on Triton

The DeepSeek pair used 192 GB200 GPUs with the same TP16/PP1 non-colocated
topology and the same Steps 3-9,11-19 window.

Component	Triton	FlashInfer TRT-LLM	FlashInfer delta
Generation	21.529s	16.876s	-4.654s (-21.62%)
Policy training	7.365s	7.265s	-0.100s (-1.36%)
Policy/reference logprobs	4.416s	4.384s	-0.031s (-0.71%)
Prepare for generation	5.760s	11.394s	+5.634s (+97.82%)
E2E step time	40.281s	41.112s	+0.831s (+2.06%)

prepare_for_generation/transfer_and_update_weights is nested inside the
prepare phase and increased from 6.136s to 12.153s on the 15 steps where
an update occurred. Profiling attributes the added refit cost to the native
TRT-LLM reload path: expert-layout conversion, cloning deferred tensors away
from reusable transport buffers, and backend-specific weight loading. The
+5.634s prepare regression exceeds the -4.654s generation saving, while
policy training and logprobs are effectively unchanged. Therefore raw E2E is
2.06% slower. FlashInfer generated 2.45% more tokens, so pooled E2E
throughput was at parity (+0.05%), but that is not evidence for changing the
recipe default.

W&B: DeepSeek FlashInfer,
DeepSeek Triton.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[seonjinn]

/ok to test 999ea23

[seonjinn]

/ok to test c228a56

[seonjinn]

/ok to test a5b8645

[seonjinn]

Contributor self-review refresh for eb1d3f2d71007687b40a09f5611252fe3e3c459d after the latest main merge.

The recipe-only backend/parallelism changes were rechecked against current schemas and their #3659/#3669 runtime dependency. Recipe minimization and Ruff/format/config checks pass locally, with no new source-level findings.

[seonjinn]

/ok to test 1670a5f