[Perf][Kernel] Fused SiLU+Mul+Quant kernel for NVFP4 cutlass_moe

[mgoin]

Purpose

We can easily fuse the silu_and_mul operation into the nvfp4 quantization operation for moe, which we already do for dense nvfp4 as of #23671. This PR just generalizes some utils to expose the same functionality for expert quantization. This seems to improve latency by ~2% and throughput by ~4%

Before:

After:

Test Plan

Benchmarks and evals

Test Result

Latency Benchmark

vllm serve nvidia/Qwen3-30B-A3B-NVFP4
vllm bench serve --input-len 100 --output-len 100 --num-prompts 8

# MAIN
============ Serving Benchmark Result ============
Successful requests:                     8         
Failed requests:                         0         
Benchmark duration (s):                  0.80      
Total input tokens:                      800       
Total generated tokens:                  800       
Request throughput (req/s):              10.06     
Output token throughput (tok/s):         1005.70   
Peak output token throughput (tok/s):    800.00    
Peak concurrent requests:                8.00      
Total token throughput (tok/s):          2011.41   
---------------Time to First Token----------------
Mean TTFT (ms):                          33.40     
Median TTFT (ms):                        34.42     
P99 TTFT (ms):                           40.03     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.62      
Median TPOT (ms):                        7.62      
P99 TPOT (ms):                           7.62      
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.62      
Median ITL (ms):                         7.67      
P99 ITL (ms):                            8.22      
==================================================

# PR
============ Serving Benchmark Result ============
Successful requests:                     8         
Failed requests:                         0         
Benchmark duration (s):                  0.78      
Total input tokens:                      800       
Total generated tokens:                  800       
Request throughput (req/s):              10.25     
Output token throughput (tok/s):         1025.21   
Peak output token throughput (tok/s):    800.00    
Peak concurrent requests:                8.00      
Total token throughput (tok/s):          2050.42   
---------------Time to First Token----------------
Mean TTFT (ms):                          31.20     
Median TTFT (ms):                        32.42     
P99 TTFT (ms):                           34.46     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.53      
Median TPOT (ms):                        7.53      
P99 TPOT (ms):                           7.53      
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.53      
Median ITL (ms):                         7.55      
P99 ITL (ms):                            8.30      
==================================================

Throughput Benchmark

vllm serve nvidia/Qwen3-30B-A3B-NVFP4
vllm bench serve --input-len 100 --output-len 100 --num-prompts 512

# MAIN
============ Serving Benchmark Result ============
Successful requests:                     512       
Failed requests:                         0         
Benchmark duration (s):                  2.65      
Total input tokens:                      51200     
Total generated tokens:                  51200     
Request throughput (req/s):              193.35    
Output token throughput (tok/s):         19335.14  
Peak output token throughput (tok/s):    30913.00  
Peak concurrent requests:                512.00    
Total token throughput (tok/s):          38670.29  
---------------Time to First Token----------------
Mean TTFT (ms):                          812.20    
Median TTFT (ms):                        793.07    
P99 TTFT (ms):                           1031.71   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.91     
Median TPOT (ms):                        18.14     
P99 TPOT (ms):                           20.28     
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.93     
Median ITL (ms):                         16.15     
P99 ITL (ms):                            87.56     
================================================== 

# PR
============ Serving Benchmark Result ============
Successful requests:                     512       
Failed requests:                         0         
Benchmark duration (s):                  2.55      
Total input tokens:                      51200     
Total generated tokens:                  51200     
Request throughput (req/s):              200.45    
Output token throughput (tok/s):         20044.52  
Peak output token throughput (tok/s):    32996.00  
Peak concurrent requests:                512.00    
Total token throughput (tok/s):          40089.04  
---------------Time to First Token----------------
Mean TTFT (ms):                          776.44    
Median TTFT (ms):                        759.85    
P99 TTFT (ms):                           1005.40   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.20     
Median TPOT (ms):                        17.63     
P99 TPOT (ms):                           19.88     
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.24     
Median ITL (ms):                         15.43     
P99 ITL (ms):                            100.69    
==================================================

Eval

vllm serve nvidia/Qwen3-30B-A3B-NVFP4
python tests/evals/gsm8k/gsm8k_eval.py --port 8000

# MAIN
Results:
Accuracy: 0.886
Invalid responses: 0.001
Total latency: 20.575 s
Questions per second: 64.106
Total output tokens: 153861
Output tokens per second: 7477.995

# PR
Results:
Accuracy: 0.883
Invalid responses: 0.000
Total latency: 20.371 s
Questions per second: 64.748
Total output tokens: 153967
Output tokens per second: 7558.026

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a fused kernel, silu_and_mul_scaled_fp4_experts_quant, to optimize the performance of Mixture of Experts (MoE) layers using NVFP4 quantization. The fusion is controlled by a new FUSE_SILU_MUL template parameter, and the changes are well-integrated into the existing kernel infrastructure. The PR also includes beneficial refactoring, such as centralizing input validation logic. The benchmark results provided in the description demonstrate a clear performance improvement. My review includes one suggestion to improve C++ code quality by using const references for read-only parameters, which is a good practice for correctness and can aid compiler optimizations. Overall, this is a solid performance enhancement.

[yewentao256]

Nice work! Just a few small thoughts

[yewentao256]

here low_latency seems not used?

[mgoin]

Yeah... I'm not sure, I didn't touch this code so I don't want to change that in this PR

[mgoin]

Yeah... I'm not sure, I didn't touch this code so I don't want to change that in this PR

[yewentao256]

LGTM, thanks for the work!