[Model]Fuse MiniMax-M3 dense-layer KV-cache insert into qknorm+rope k…

[lengrongfu]

Purpose

Dense layers previously ran the fused qknorm+rope kernel without cache args and let the generic Attention layer insert k/v via a separate reshape_and_cache_flash, costing an extra full HBM round-trip of k/v and a kernel launch per dense layer (the majority of M3 layers). The device kernel already supports insert without the index branch; this enables it end to end:

• csrc: instantiate the dense+insert launch (kHasIndex=false, kInsertKV=true) and drop the host check that forbade it.
• Attention: add skip_kv_cache_update_by_layer (default False) so a model layer that inserts k/v itself can skip unified_kv_cache_update while still passing k/v to the backend impl.
• MiniMaxM3Attention (nvidia only): insert k/v through the fused kernel when the backend runs the KV update as a separate step (forward_includes_kv_cache_update=False); all such backends keep the paged cache in the [nb, nkv, bs, 2*hd] layout the kernel writes, and its host checks fail loudly otherwise. Falls back to the unfused path on memory-profiling runs (no slot mapping).

Test Plan

Test Result

• Benc command: vllm bench serve --served-model-name /new-model/MiniMax-M3-MXFP8/ --model /new-model/MiniMax-M3-MXFP8/ --tokenizer /new-model/MiniMax-M3-MXFP8/ --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input 1000 --random-output 100 --seed 130 --base-url http://localhost:8000 --num-prompts 10
• Baseline

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Benchmark duration (s):                  16.23     
Total input tokens:                      11760     
Total generated tokens:                  1000      
Request throughput (req/s):              0.62      
Output token throughput (tok/s):         61.60     
Peak output token throughput (tok/s):    70.00     
Peak concurrent requests:                10.00     
Total token throughput (tok/s):          785.98    
---------------Time to First Token----------------
Mean TTFT (ms):                          1022.31   
Median TTFT (ms):                        1071.78   
P99 TTFT (ms):                           1250.09   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          153.17    
Median TPOT (ms):                        152.94    
P99 TPOT (ms):                           157.94    
---------------Inter-token Latency----------------
Mean ITL (ms):                           151.64    
Median ITL (ms):                         152.75    
P99 ITL (ms):                            155.78    
==================================================

• Optimization bench data

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Benchmark duration (s):                  16.09     
Total input tokens:                      11760     
Total generated tokens:                  1000      
Request throughput (req/s):              0.62      
Output token throughput (tok/s):         62.15     
Peak output token throughput (tok/s):    77.00     
Peak concurrent requests:                10.00     
Total token throughput (tok/s):          793.03    
---------------Time to First Token----------------
Mean TTFT (ms):                          1020.01   
Median TTFT (ms):                        1069.64   
P99 TTFT (ms):                           1246.90   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          151.74    
Median TPOT (ms):                        151.50    
P99 TPOT (ms):                           156.53    
---------------Inter-token Latency----------------
Mean ITL (ms):                           150.22    
Median ITL (ms):                         151.40    
P99 ITL (ms):                            154.28    
==================================================

• gsm8k test Baseline

Results:
Accuracy: 0.879
Invalid responses: 0.000
Total latency: 121.214 s
Questions per second: 10.882
Total output tokens: 118606
Output tokens per second: 978.483

• gsm8k test Current PR

Results:
Accuracy: 0.881
Invalid responses: 0.000
Total latency: 125.443 s
Questions per second: 10.515
Total output tokens: 118157
Output tokens per second: 941.916

While ensuring no loss of progress, the optimized TPOT and TTFT show an average performance improvement of 2ms.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[lengrongfu]

@yewentao256 Hi, can you help review this pr, current pr fused dense layer kv-cache insert into qknorm kernel.

[yewentao256]

Thanks for the work, there are only 3 dense layers out of 60 for minimax m3, I don't think it will benefit a lot for e2e throughput/latency, and not worth the complexity we introduce here.