[Monishver11]

Purpose

silu_and_mul_quant (act_and_mul_quant_kernel in csrc/quantization/activation_kernels.cu) is the fused SiLU-and-mul + static per-tensor FP8 quant that the manual activation+quant fusion (#51415, maybe_fused_act_quant) routes static-FP8 models through by default.

Its 128-bit vectorized loop has never executed. The loop bound is computed as round_to_previous_multiple_of(elems_per_128bit_load, block_end), with the two arguments swapped: the helper rounds its first argument down to a multiple of its second, so this rounds 8 down to a multiple of block_end, which is 0 for any d > 8. With vec_loop_end = 0 the vectorized loop has an empty range and every element goes through the scalar cleanup loop with 2-byte loads and 1-byte stores. Present since the kernel landed in #10867. Runtime proof on main: with the input and output pointers misaligned by 8 bytes the op completes and matches the aligned result for d = 14336 and 10752 (no 16-byte load executed), and faults with misaligned address at d = 8, the one shape where the bound is non-zero.

The cost: on H100 the kernel runs at 1.9 TB/s effective against 3.1 TB/s for the Inductor-fused kernel it replaced, and Llama-3.1-8B-FP8 serves about 3.5% slower with the fusion wired than on the Inductor path (numbers below).

This PR:

• Fixes the argument order, so the bound is block_end rounded down to the vector width and the 128-bit loop covers the block.
• Takes the vectorized path only when d is a multiple of the vector width (8 for bf16/fp16) and both data pointers are 16-byte aligned, via the existing vllm::is_16byte_aligned. Rows of other widths are not 16-byte aligned (d = 781, hidden size 1562 in the unit test, starts odd rows 4 bytes off) and would fault on the fixed loop; they keep the scalar loop, which is what every shape runs today.
• In the scalar case the cleanup loop starts at the block's own block_start instead of 0. With 2 to 4 blocks per row at 32 tokens or fewer, starting at 0 made each block redo the preceding blocks' elements (same values written, wasted bandwidth).

Output is bit-identical to main: both loops evaluate the same per-element expression. Verified below. No change to grid, block size or arithmetic; the remaining gap to the Inductor kernel (0.9x at prefill, about 0.5x at decode) is the subject of a separate follow-up.

Not a duplicate. No open PR touches this kernel. #49828 adds a separate per-token dynamic kernel in the same file with its own launch. #59514 is my earlier PR that bundled this fix with a kernel rewrite; it is being split into this fix and a follow-up so each can be reviewed on its own.

Test Plan

• Runtime proof of the dead loop on main and of the alignment guard on this branch (misaligned-pointer test, script in the attached logs).
• Bitwise vs main (4b61f30): sha256 of the output bytes over bf16 and fp16, tokens 1 to 16384, d in {8192, 10752, 14336, 28672} plus {8, 24, 781, 2048, 4104, 10760, 14341}, and misaligned-pointer cases; 302 cases.
• pytest tests/kernels/test_fused_quant_activation.py tests/compile/passes/test_silu_mul_quant_manual_fusion.py tests/compile/passes/test_silu_mul_quant_fusion.py tests/compile/passes/test_functionalization.py
• Microbenchmark: H100 80GB, driver 615, CUDA 13.0, torch 2.14.0+cu130, do_bench_cudagraph; the op on both builds and a torch.compile of SiluAndMul.forward_native + QuantFP8(static).forward_native as the Inductor column.
• E2e on RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 with main's own wiring, three interleaved arms: main kernel, this fix, and the same tree with the llama.py wiring line reverted (Inductor path):

lm_eval --model vllm --model_args pretrained=RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8,max_model_len=4096,gpu_memory_utilization=0.9,seed=0 --tasks gsm8k --num_fewshot 5 --batch_size auto --seed 0
vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --max-model-len 2048
vllm bench serve --backend vllm --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 1000 --request-rate inf --ignore-eos --seed 0
vllm bench latency --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --max-model-len 2048 --batch-size 32 --input-len 1024 --output-len 32 --profile

Test Result

• Proof: on main, d = 14336 and 10752 complete on 8-byte-misaligned pointers and match the aligned output, d = 8 faults with CUDA error: misaligned address. On this branch all three complete and match (the guard sends misaligned inputs to the scalar loop).
• Bitwise vs main: 302 identical, 0 mismatched.
• Tests: 263 passed, 76 skipped (4 files, unchanged).

Microbenchmark, bf16, d = 14336 (us):

tokens	main	fix	Inductor	fix vs main	fix vs Inductor
1	9.01	2.30	1.29	3.91x	0.56x
32	9.34	3.45	1.75	2.71x	0.51x
256	11.03	8.53	4.90	1.29x	0.57x
1024	54.50	30.27	25.63	1.80x	0.85x
4096	195.15	106.43	95.54	1.83x	0.90x
16384	747.86	407.32	375.24	1.84x	0.92x

Geomean 2.18x vs main over 32 configs (tokens 1 to 16384, four d values), range 1.25x to 5.07x. Against Inductor: 0.34x to 0.66x at 256 tokens and below, 0.65x to 0.96x from 1024 tokens up (the 0.65x is 1024 tokens at d = 8192; the other prefill configs are 0.82x to 0.96x). The wider sweep in the attached logs (bf16 and fp16, 64 cases) gives 1.99x geomean, range 1.26x to 4.67x.

E2e, Llama-3.1-8B-FP8, H100

GSM8K 5-shot	flexible-extract	strict-match
main	0.7817 ± 0.0114	0.7089 ± 0.0125
fix	0.7817 ± 0.0114	0.7089 ± 0.0125

Identical, as expected from bit-identical kernel output.

vllm bench serve, three interleaved rounds per arm:

metric	main	fix	Inductor path	fix vs main	fix vs Inductor
Request throughput (req/s)	37.76	38.75	39.15	+2.6%	-1.0%
Output token throughput (tok/s)	4834	4960	5011	+2.6%	-1.0%
Mean TTFT (ms)	11708	11373	11271	-2.9%	+0.9%
Mean TPOT (ms)	71.80	69.97	69.22	-2.6%	+1.1%

Per-round req/s: main 37.82 / 37.77 / 37.70; fix 38.68 / 38.55 / 39.02; Inductor 39.01 / 39.32 / 39.12. The kernel was switched by rebuilding between rounds. Every fix round is above every main round and below every Inductor round.

Profile, one vllm bench latency iteration (batch 32, 1024/32), 1088 launches of the fused op per arm:

arm	fused op	share of GPU time	decode launch (median)	prefill launch (mean)
main	68.94 ms	10.1%	15.33 us	809 us
fix	36.60 ms	5.6%	4.03 us	494 us
Inductor path	26.96 ms	4.1%	2.02 us	377 us

Full logs attached.

I used Claude Code for this change. I reviewed every line and ran the tests and benchmarks above myself.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR (bug fix: the vectorized loop never ran; output unchanged).
• The test plan (runtime proof, bitwise, unit tests, microbenchmark, e2e).
• The test results (tables above).
• (Optional) Documentation update: n/a.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Monishver11]

Gist: the vectorized loop in silu_and_mul_quant never ran because the two arguments of round_to_previous_multiple_of were swapped, so every element went through the scalar path. Output is bit-identical to main.

This was the first part of #59514, which also rewrote the kernel. I split it so this PR carries only the fix, keeping it minimal and easy to review.

On H100, the kernel is 2.18x faster than main (geomean over 32 configs), and Llama-3.1-8B-FP8 serving goes from 37.76 to 38.75 req/s (+2.6%), 1.0% below the Inductor path (I have tested further changes, a flat grid and an fp32 fast-math chain, that close this gap and get slightly ahead of Inductor; they would be a follow-up PR).

Output logs: silu_mul_quant_fixA_logs.tgz

@mgoin, @yewentao256, can you pls review this when you get some time?

[yewentao256]

Thanks for the work!

Please do the following:

• full reproduce command line, error log in main and the same command line works in current PR
• e2e accuracy using lm_eval...
• e2e benchmark using vllm bench serve...

When doing reproduce / e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

pytest tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# main
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1070, in __init__
  | (EngineCore pid=1580)     super().__init__(
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 160, in __init__
  | (EngineCore pid=1580)     self.scheduler: SchedulerInterface = Scheduler(
  | (EngineCore pid=1580)                                          ^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
  | (EngineCore pid=1580)     super().__init__(*args, **kwargs)
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 277, in __init__
  | (EngineCore pid=1580)     self.kv_cache_manager = KVCacheManager(
  | (EngineCore pid=1580)                             ^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 151, in __init__
  | (EngineCore pid=1580)     self.coordinator = get_kv_cache_coordinator(
  | (EngineCore pid=1580)                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 891, in get_kv_cache_coordinator
  | (EngineCore pid=1580)     return HybridKVCacheCoordinator(
  | (EngineCore pid=1580)            ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 563, in __init__
  | (EngineCore pid=1580)     assert all(
  | (EngineCore pid=1580)            ^^^^
  | (EngineCore pid=1580) AssertionError: Each KV cache group's real block_size must be divisible by hash_block_size. block_sizes=[544, 544, 544, 544, 181], hash_block_size=98464
  | FAILED
  | v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2

# Now
================= 1 passed, 23 warnings in 174.57s (0:02:54) ==================

"lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9212|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9212|±  |0.0074|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9151|±  |0.0077|
|     |       |strict-match    |     5|exact_match|↑  |0.9158|±  |0.0076|

`lm_eval run --model local-chat-completions --model_args base_url=http://127.0.0.1:9256/v1/chat/completions,model=$MODEL,num_concurrent=1024,seed=256 --tasks aime25 --trust_remote_code`
# now
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|
# main
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|

vllm bench serve --model zai-org/GLM-5.2-FP8  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16

# Now

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.08      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              41.50     
Output token throughput (tok/s):         41.50     
Peak output token throughput (tok/s):    76.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          340041.81 
---------------Time to First Token----------------
Mean TTFT (ms):                          1669.15   
Median TTFT (ms):                        1669.02   
P99 TTFT (ms):                           2724.38   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.23      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              39.63     
Output token throughput (tok/s):         39.63     
Peak output token throughput (tok/s):    50.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          324671.20 
---------------Time to First Token----------------
Mean TTFT (ms):                          1668.69   
Median TTFT (ms):                        1649.00   
P99 TTFT (ms):                           3126.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

[Monishver11]

Thanks for the review @yewentao256. Raw logs below (all from the tgz attached in my comment above).

Reproduce: the bug is a dead code path, so the probe is scripts/proof_main.py (in the tgz). It runs the op with the input and output pointers deliberately misaligned by 8 bytes. The vectorized loop uses 16-byte loads, and a 16-byte load on a misaligned pointer faults with a misaligned address. So on main, if the op completes, the vectorized loop did not run. It completes for d = 14336 and 10752, and faults only at d = 8, the one shape where the swapped bound is non-zero. On this PR, all three complete, because misaligned pointers now take the scalar loop.

In the output, A is the real shapes (d = 14336, 10752) where the swapped bound is 0, and B is d = 8, the one shape where the bound is non-zero, and the loop actually executes.

python proof_main.py

# main
== A) d=14336 and d=10752, 36 tokens, pointers misaligned by 8 bytes (must succeed)
d=14336 tokens=36 input_ptr%16=8 out_ptr%16=4
d=14336: op completed on misaligned pointers, equal to aligned result: True
exit 0 
d=10752 tokens=36 input_ptr%16=8 out_ptr%16=4
d=10752: op completed on misaligned pointers, equal to aligned result: True
exit 0 
== B) d=8, 36 tokens, same misalignment (must fault with 'misaligned address')
d=8 tokens=36 input_ptr%16=8 out_ptr%16=4
exit 1
stderr tail: s.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
The CUDA driver logged these messages, which may provide useful details:
Returning 716 (CUDA_ERROR_MISALIGNED_ADDRESS) from cuCtxSynchronize_v2
PROOF_MAIN OK (d=8 faulted on a 16-byte load; d=14336/10752 did not)

# Now
== A) d=14336 and d=10752, 36 tokens, pointers misaligned by 8 bytes (must succeed)
d=14336 tokens=36 input_ptr%16=8 out_ptr%16=4
d=14336: op completed on misaligned pointers, equal to aligned result: True
exit 0 
d=10752 tokens=36 input_ptr%16=8 out_ptr%16=4
d=10752: op completed on misaligned pointers, equal to aligned result: True
exit 0 
== B) d=8, 36 tokens, same misalignment (must fault with 'misaligned address')
d=8 tokens=36 input_ptr%16=8 out_ptr%16=4
d=8: op completed on misaligned pointers, equal to aligned result: True
exit 0
stderr tail: W1014 23:08:23.773000 28175 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead
W1014 23:08:23.774000 28175 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead
W1014 23:08:23.774000 28175 torch/_library/opaque_object.py:211] typ='value' is deprecated, use typ='constant' instead
PROOF_MAIN INCONCLUSIVE (d=8 faulted on a 16-byte load; d=14336/10752 did not)

The last line is the script's verdict for the main run (it checks whether d = 8 faults); on this branch the absence of a fault is the intended result.

lm_eval --model vllm --model_args pretrained=RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8,max_model_len=4096,gpu_memory_utilization=0.9,seed=0 --tasks gsm8k --num_fewshot 5 --batch_size auto --seed 0

# main
vllm ({'pretrained': 'RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8', 'max_model_len': 4096, 'gpu_memory_utilization': 0.9, 'seed': 0}), gen_kwargs: ({}), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7817|±  |0.0114|
|     |       |strict-match    |     5|exact_match|↑  |0.7089|±  |0.0125|

# Now
vllm ({'pretrained': 'RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8', 'max_model_len': 4096, 'gpu_memory_utilization': 0.9, 'seed': 0}), gen_kwargs: ({}), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7817|±  |0.0114|
|     |       |strict-match    |     5|exact_match|↑  |0.7089|±  |0.0125|

vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --max-model-len 2048
vllm bench serve --backend vllm --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 1000 --request-rate inf --ignore-eos --seed 0
# three interleaved rounds per build

# main, round 1
============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  26.44     
Total input tokens:                      1024000   
Total generated tokens:                  128000    
Request throughput (req/s):              37.82     
Output token throughput (tok/s):         4841.05   
Peak output token throughput (tok/s):    14661.00  
Peak concurrent requests:                1000.00   
Total token throughput (tok/s):          43569.49  
---------------Time to First Token----------------
Mean TTFT (ms):                          11694.52  
Median TTFT (ms):                        12713.09  
P99 TTFT (ms):                           24592.84  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          71.71     
Median TPOT (ms):                        82.73     
P99 TPOT (ms):                           101.96    
---------------Inter-token Latency----------------
Mean ITL (ms):                           71.77     
Median ITL (ms):                         36.61     
P99 ITL (ms):                            240.58    
==================================================

# Now, round 1
============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  25.86     
Total input tokens:                      1024000   
Total generated tokens:                  128000    
Request throughput (req/s):              38.68     
Output token throughput (tok/s):         4950.49   
Peak output token throughput (tok/s):    14747.00  
Peak concurrent requests:                1000.00   
Total token throughput (tok/s):          44554.42  
---------------Time to First Token----------------
Mean TTFT (ms):                          11443.56  
Median TTFT (ms):                        12448.30  
P99 TTFT (ms):                           24083.44  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          69.96     
Median TPOT (ms):                        80.43     
P99 TPOT (ms):                           100.15    
---------------Inter-token Latency----------------
Mean ITL (ms):                           69.98     
Median ITL (ms):                         36.12     
P99 ITL (ms):                            241.35    
==================================================

# main, round 2
============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  26.48     
Total input tokens:                      1024000   
Total generated tokens:                  128000    
Request throughput (req/s):              37.77     
Output token throughput (tok/s):         4834.75   
Peak output token throughput (tok/s):    14592.00  
Peak concurrent requests:                1000.00   
Total token throughput (tok/s):          43512.73  
---------------Time to First Token----------------
Mean TTFT (ms):                          11691.02  
Median TTFT (ms):                        12751.91  
P99 TTFT (ms):                           24667.18  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          71.84     
Median TPOT (ms):                        82.70     
P99 TPOT (ms):                           102.88    
---------------Inter-token Latency----------------
Mean ITL (ms):                           71.88     
Median ITL (ms):                         36.75     
P99 ITL (ms):                            250.28    
==================================================

# Now, round 2
============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  25.94     
Total input tokens:                      1024000   
Total generated tokens:                  128000    
Request throughput (req/s):              38.55     
Output token throughput (tok/s):         4934.94   
Peak output token throughput (tok/s):    14644.00  
Peak concurrent requests:                1000.00   
Total token throughput (tok/s):          44414.45  
---------------Time to First Token----------------
Mean TTFT (ms):                          11436.11  
Median TTFT (ms):                        12455.63  
P99 TTFT (ms):                           24066.09  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          70.19     
Median TPOT (ms):                        80.64     
P99 TPOT (ms):                           99.98     
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.22     
Median ITL (ms):                         36.72     
P99 ITL (ms):                            218.51    
==================================================

# main, round 3
============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  26.52     
Total input tokens:                      1024000   
Total generated tokens:                  128000    
Request throughput (req/s):              37.70     
Output token throughput (tok/s):         4825.80   
Peak output token throughput (tok/s):    14520.00  
Peak concurrent requests:                1000.00   
Total token throughput (tok/s):          43432.22  
---------------Time to First Token----------------
Mean TTFT (ms):                          11737.78  
Median TTFT (ms):                        12696.91  
P99 TTFT (ms):                           24654.12  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          71.86     
Median TPOT (ms):                        82.64     
P99 TPOT (ms):                           102.31    
---------------Inter-token Latency----------------
Mean ITL (ms):                           71.93     
Median ITL (ms):                         36.32     
P99 ITL (ms):                            229.70    
==================================================

# Now, round 3
============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  25.63     
Total input tokens:                      1024000   
Total generated tokens:                  128000    
Request throughput (req/s):              39.02     
Output token throughput (tok/s):         4994.79   
Peak output token throughput (tok/s):    14387.00  
Peak concurrent requests:                1000.00   
Total token throughput (tok/s):          44953.15  
---------------Time to First Token----------------
Mean TTFT (ms):                          11240.41  
Median TTFT (ms):                        12215.92  
P99 TTFT (ms):                           23798.15  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          69.76     
Median TPOT (ms):                        80.20     
P99 TPOT (ms):                           99.57     
---------------Inter-token Latency----------------
Mean ITL (ms):                           69.81     
Median ITL (ms):                         34.77     
P99 ITL (ms):                            235.19    
==================================================

[yewentao256]

LGTM, thanks for the work!

[yewentao256]

/ci run

[github-actions]

✅ @Monishver11, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[github-actions]

✅ Triggered Buildkite CI #95155 for commit 7db867f57141.

ℹ️ This PR is 56 commits behind upstream main. Merged upstream main into this branch as 7db867f57141 before starting CI. Pull the branch before pushing more changes.

[Monishver11]

/ci retry

[github-actions]

✅ Queued 2 failed job(s) for retry in Buildkite CI #95155.

[Monishver11]

Hi @yewentao256, CI is green except nvidia-h200-mig-35gb-quantization-shard-4/4, which failed the same way on the retry: a GPU MMU fault in hadacore::hadamard_transform_kernel during tests/quantization/test_per_token_kv_cache.py::test_reshape_and_cache_per_token_head[int4-0-16-128-1-42]. It is not related to this change(nothing in the PR touches hadacore or the int4 KV path), and #61700's build crashes at the same test case. Could you pls check whether it also fails on main?

[Monishver11]

Thanks @yewentao256 for the review and @khluu for the merge.