Enable flashinfer::trtllm_allreduce_fusion with PDL

[b8zhong]

Motivation

For BS = 1:

With PDL:

Thus, the latency of AR can be (nearly) fully hidden here.

Modifications

With regards to read after write hazards in the dependent kernel, it will only happen when:

PDL is issued without griddepcontrol.wait, which never is in the case in: fp4_quantize in all backends in Flashinfer, and fused_a_gemm. If there is no PDL signal (like in per token quant 8bit v2), then it will fully serialize like before (for ex, see the cvt_fp16_to_fp4 in the sgl-kernel one, which does not issue either PDL or GDC await. However, when we switch to #23745 (comment) for quantize, it will work properly.

Accuracy Tests

python3 -m sglang.launch_server \
  --model-path nvidia/DeepSeek-V3.2-NVFP4 \
  --tp 4 \
  --ep 4 \
  --quantization modelopt_fp4 \
  --tool-call-parser deepseekv32 \
  --reasoning-parser deepseek-v3 \
  --port 30020 \
  --cuda-graph-bs 1 2 4 8 16 32 64 128 160 192 224 256 288 320 352 384 416 448 480 512 \
  --max-running-requests 512

python3 -m sglang.test.run_eval --port 30020 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3

Repeat: 8, mean: 0.834██████████████████████████████████████████████████████████████████████████████████████████████████████████████▌       | 187/198 [25:35<00:50,  4.55s/it]
Scores: ['0.813', '0.859', '0.869', '0.813', '0.869', '0.798', '0.843', '0.808']

Speed Tests and Profiling

109 -> 111.20 tps

[gemini-code-assist]

Code Review

This pull request updates the flashinfer_allreduce_residual_rmsnorm function in flashinfer_comm_fusion.py by adding the trigger_completion_at_end argument to an internal call. The review feedback correctly points out that this argument is hardcoded to False, which ignores the value provided via the function's parameter, and suggests passing the parameter variable instead.

[nvpohanh]

Thanks! LGTM
cc @nvjullin @wenscarl