[Perf][MXFP4] Optimize activation quant for mxfp4. 3x kernel speedup, 5% throughput improvement by ilmarkov · Pull Request #63101 · vllm-project/vllm

ilmarkov · 2026-10-22T10:55:54Z

Overview

Pass optional sorted_expert_ids through Cutlass MXFP4 MoE activation quantization so each sorted token row uses a direct expert index instead of searching expert offset tables.

Claims

Kernel: sorted_expert_id path is bitwise identical to offset_search on microbenchmark cases; up to ~3.3× lower median latency at 256 experts for representative input / intermediate_fused shapes.
E2E: On RedHatAI/GLM-5.3-MXFP4 (TP=4, no spec), ~+5.7% output tok/s vs main.
API change: Optional sorted_expert_ids on mxfp4_experts_quant, silu_and_mul_mxfp4_experts_quant, and get_cutlass_moe_mm_data; wired from Cutlass MXFP4 MoE when routing metadata exists.

Validation

Microbenchmark

benchmarks/kernels/benchmark_mxfp4_experts_activation_quant.py
Hardware: NVIDIA B300

kernel only

Kernel	E=16	E=32	E=64	E=128	E=256
mxfp4 main → PR (µs)	1.94 → 1.98	2.61 → 2.19 (1.19×)	3.29 → 2.19 (1.50×)	4.62 → 2.16 (2.14×)	7.33 → 2.16 (3.39×)
silu+mul main → PR	2.48 → 2.44 (1.02×)	2.92 → 2.44 (1.20×)	3.56 → 2.44 (1.46×)	4.85 → 2.43 (1.99×)	7.44 → 2.43 (3.07×)

mm_data + shuffle + quant

Kernel	E=16	E=32	E=64	E=128	E=256
mxfp4 main → PR (µs)	10.55 → 10.77	11.85 → 11.61 (1.02×)	14.06 → 12.96 (1.08×)	17.56 → 14.94 (1.18×)	22.48 → 16.78 (1.34×)
silu+mul main → PR	10.18 → 10.41	11.45 → 11.27 (1.02×)	13.70 → 11.42 (1.20×)	17.32 → 14.65 (1.18×)	22.30 → 16.46 (1.35×)

Benchmarks time get_cutlass_moe_mm_data + activation quant together; main and PR's paths share ~14–15 µs metadata at 256 experts (mo overhead on sorted_expert_ids), so the reported speedup is ~1.4× on the full replay while quant still saves ~5–6 µs by skipping offset lookup.

E2E benchmark

vllm serve:

vllm serve RedHatAI/GLM-5.3-MXFP4 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --max-model-len 1048576 \
  --served-model-name RedHatAI/GLM-5.3-MXFP4 \
  --kv-cache-dtype fp8_e4m3 \
  --gpu-memory-utilization 0.83 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 10 \
  --enable-prefix-caching \
  --disable-custom-all-reduce \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --kernel-config.enable_flashinfer_autotune=False

vllm bench serve \
  --model RedHatAI/GLM-5.3-MXFP4 \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input-len 256 \
  --random-output-len 1024 \
  --num-prompts 32 \
  --max-concurrency 1 \
  --request-rate inf \
  --seed 42 \
  --temperature 0 \
  --ignore-eos \
  --num-warmups 2 \
  --save-result

main:

============ Serving Benchmark Result ============
Successful requests:                     32        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  383.03    
Total input tokens:                      8192      
Total generated tokens:                  32768     
Request throughput (req/s):              0.08      
Output token throughput (tok/s):         85.55     
Peak output token throughput (tok/s):    89.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          106.94    
---------------Time to First Token----------------
Mean TTFT (ms):                          274.60    
Median TTFT (ms):                        272.08    
P99 TTFT (ms):                           296.04    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.43     
Median TPOT (ms):                        11.43     
P99 TPOT (ms):                           11.44     
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.43     
Median ITL (ms):                         11.42     
P99 ITL (ms):                            11.94     
==================================================

PR:

============ Serving Benchmark Result ============
Successful requests:                     32        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  358.07    
Total input tokens:                      8192      
Total generated tokens:                  32768     
Request throughput (req/s):              0.09      
Output token throughput (tok/s):         91.51     
Peak output token throughput (tok/s):    96.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          114.39    
---------------Time to First Token----------------
Mean TTFT (ms):                          272.62    
Median TTFT (ms):                        268.01    
P99 TTFT (ms):                           297.79    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.67     
Median TPOT (ms):                        10.67     
P99 TPOT (ms):                           10.70     
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.67     
Median ITL (ms):                         10.66     
P99 ITL (ms):                            11.24     
==================================================

Gsm8k validation

Configuration	flexible-extract	strict-match
main	0.9052	0.9052
This PR	0.9060	0.9083

Details

Motivation: After MoE routing, activation rows are grouped by expert. The original path resolves each row’s expert via offset-table search; when sorted_expert_ids is already available from Cutlass MoE metadata, that search is redundant and scales poorly with expert count (microbench ~7 µs → ~2.2 µs at 256 experts).
Design: sorted_expert_ids is optional; when omitted, behavior matches offset_search.

Possible follow-up: remove redundant shared-memory expert_offsets staging in the large-m_topk MXFP4 quant kernel when sorted_expert_ids is set, and extend the microbenchmark to prefill-sized batches that hit that kernel variant.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Signed-off-by: ilmarkov <markovilya197@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: ilmarkov <markovilya197@gmail.com>

yewentao256

Thanks for the work!

Please do the following:

shrink the diff, idealy < 500 LOC including the tests
full reproduce command line, error log in main and the same command line works in current PR
e2e accuracy using lm_eval...
e2e benchmark using vllm bench serve...

When doing reproduce / e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

pytest tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# main
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1070, in __init__
  | (EngineCore pid=1580)     super().__init__(
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 160, in __init__
  | (EngineCore pid=1580)     self.scheduler: SchedulerInterface = Scheduler(
  | (EngineCore pid=1580)                                          ^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
  | (EngineCore pid=1580)     super().__init__(*args, **kwargs)
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 277, in __init__
  | (EngineCore pid=1580)     self.kv_cache_manager = KVCacheManager(
  | (EngineCore pid=1580)                             ^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 151, in __init__
  | (EngineCore pid=1580)     self.coordinator = get_kv_cache_coordinator(
  | (EngineCore pid=1580)                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 891, in get_kv_cache_coordinator
  | (EngineCore pid=1580)     return HybridKVCacheCoordinator(
  | (EngineCore pid=1580)            ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 563, in __init__
  | (EngineCore pid=1580)     assert all(
  | (EngineCore pid=1580)            ^^^^
  | (EngineCore pid=1580) AssertionError: Each KV cache group's real block_size must be divisible by hash_block_size. block_sizes=[544, 544, 544, 544, 181], hash_block_size=98464
  | FAILED
  | v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2

# Now
================= 1 passed, 23 warnings in 174.57s (0:02:54) ==================

"lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9212|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9212|±  |0.0074|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9151|±  |0.0077|
|     |       |strict-match    |     5|exact_match|↑  |0.9158|±  |0.0076|

`lm_eval run --model local-chat-completions --model_args base_url=http://127.0.0.1:9256/v1/chat/completions,model=$MODEL,num_concurrent=1024,seed=256 --tasks aime25 --trust_remote_code --apply_chat_template`
# now
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|
# main
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|

vllm bench serve --model zai-org/GLM-5.2-FP8  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16

# Now

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.08      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              41.50     
Output token throughput (tok/s):         41.50     
Peak output token throughput (tok/s):    76.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          340041.81 
---------------Time to First Token----------------
Mean TTFT (ms):                          1669.15   
Median TTFT (ms):                        1669.02   
P99 TTFT (ms):                           2724.38   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.23      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              39.63     
Output token throughput (tok/s):         39.63     
Peak output token throughput (tok/s):    50.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          324671.20 
---------------Time to First Token----------------
Mean TTFT (ms):                          1668.69   
Median TTFT (ms):                        1649.00   
P99 TTFT (ms):                           3126.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

Signed-off-by: ilmarkov <markovilya197@gmail.com>

LopezCastroRoberto

LGTM, thanks @ilmarkov

github-actions · 2026-10-23T11:37:35Z

✅ @ilmarkov, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

LopezCastroRoberto · 2026-10-23T11:37:45Z

/ci run

github-actions · 2026-10-23T11:39:01Z

✅ Triggered Buildkite CI #96879 for commit e7ea1a93e07f.

LopezCastroRoberto · 2026-10-23T13:24:51Z

@yewentao256 waiting for your approval too. Can you merge if it looks good to you as well?

LopezCastroRoberto · 2026-10-26T17:53:42Z

/ci run

github-actions · 2026-10-26T17:54:58Z

❌ This PR is 7 commits behind upstream main. The bot could not merge upstream main into this branch (API returned 403: user doesn't have permission to update head repository). No new CI build was started. Either merge or rebase onto the latest main yourself and rerun /ci run, or use /ci run --allow-stale to test this branch as is.

LopezCastroRoberto · 2026-10-26T17:56:09Z

/ci run

github-actions · 2026-10-26T17:56:32Z

✅ Triggered Buildkite CI #97369 for commit 21ae9b6867f7.

… 5% throughput improvement (vllm-project#63101) Signed-off-by: ilmarkov <markovilya197@gmail.com> Co-authored-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>

mergify Bot added ci/build performance Performance-related issues quantization nvidia labels Oct 22, 2026

github-project-automation Bot added this to NVIDIA Oct 22, 2026

Use sorted_experts_ids to avoid the search

2b96154

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ilmarkov force-pushed the imarkov/mxfp4-optimize-activation-conversion branch from 9ef3ff9 to 2b96154 Compare October 22, 2026 11:12

ilmarkov added 2 commits October 22, 2026 12:09

Address comments

798e259

Signed-off-by: ilmarkov <markovilya197@gmail.com>

cleanup

00b313c

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ilmarkov marked this pull request as ready for review October 22, 2026 12:45

ilmarkov requested review from bnellnm, mgoin, pavanimajety, wzhao18 and zyongye as code owners October 22, 2026 12:45

claude Bot reviewed Oct 22, 2026

View reviewed changes

Update tests and benchmarks

519f185

Signed-off-by: ilmarkov <markovilya197@gmail.com>

LopezCastroRoberto reviewed Oct 22, 2026

View reviewed changes

Comment thread csrc/quantization/fp4/mxfp4_experts_quant.cu Resolved

LopezCastroRoberto reviewed Oct 22, 2026

View reviewed changes

Comment thread benchmarks/kernels/benchmark_mxfp4_experts_activation_quant.py Resolved

yewentao256 reviewed Oct 22, 2026

View reviewed changes

ilmarkov added 2 commits October 22, 2026 16:51

Address comments. Cleanup

6c8a389

Signed-off-by: ilmarkov <markovilya197@gmail.com>

Upd benchmark

c2c8863

Signed-off-by: ilmarkov <markovilya197@gmail.com>

LopezCastroRoberto approved these changes Oct 23, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Oct 23, 2026

LopezCastroRoberto added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 23, 2026

Merge branch 'main' into imarkov/mxfp4-optimize-activation-conversion

e7ea1a9

Merge branch 'main' into imarkov/mxfp4-optimize-activation-conversion

992108f

Merge branch 'main' into imarkov/mxfp4-optimize-activation-conversion

21ae9b6

mgoin approved these changes Oct 26, 2026

View reviewed changes

mgoin merged commit f3bdb0e into vllm-project:main Oct 26, 2026
431 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Oct 26, 2026

aoshen02 mentioned this pull request Oct 28, 2026

[Model Loader] Direct reload of online-quantized modules aoshen02/vllm#190

Open

Navigation Menu

Repository navigation

[Perf][MXFP4] Optimize activation quant for mxfp4. 3x kernel speedup, 5% throughput improvement - #63101

[Perf][MXFP4] Optimize activation quant for mxfp4. 3x kernel speedup, 5% throughput improvement#63101
mgoin merged 9 commits into
vllm-project:mainfrom
neuralmagic:imarkov/mxfp4-optimize-activation-conversion

ilmarkov commented Oct 22, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Uh oh!

yewentao256 left a comment

Uh oh!

LopezCastroRoberto left a comment

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

LopezCastroRoberto commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

LopezCastroRoberto commented Oct 23, 2026

Uh oh!

LopezCastroRoberto commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

LopezCastroRoberto commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ilmarkov commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Microbenchmark

E2E benchmark

Gsm8k validation

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Uh oh!

yewentao256 left a comment

Choose a reason for hiding this comment

Uh oh!

LopezCastroRoberto left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

LopezCastroRoberto commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

LopezCastroRoberto commented Oct 23, 2026

Uh oh!

LopezCastroRoberto commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

LopezCastroRoberto commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ilmarkov commented Oct 22, 2026 •

edited

Loading