Skip to content

Add flashinfer rmsnorm + quant fusion support SM90, SM100, SM120 - #32994

Merged
BBuf merged 21 commits into
sgl-project:mainfrom
DevashishLal-CB:dev/dlal/norm-quant-fusion
Aug 4, 2026
Merged

Add flashinfer rmsnorm + quant fusion support SM90, SM100, SM120#32994
BBuf merged 21 commits into
sgl-project:mainfrom
DevashishLal-CB:dev/dlal/norm-quant-fusion

Conversation

@DevashishLal-CB

@DevashishLal-CB DevashishLal-CB commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

I had been working on the torch compile fusion approach for this for a while and am not sure what's gonna happen with it but the kernel gains are significant and can easily be done directly, this builds upon the norm quant kernels added to flashinfer which are faster than the kernels I tried writing for jit kernels (cuz they use clusters)

H100 Gains are in the range of 1.5%~2.5%
B200 Gains are in the range of 4%~5%

references

To be merged in two parts due to AOT kernel changes

B200 Gains

Metric BS4 BS8 BS16 BS32 BS64 BS128 BS256 Mean
Benchmark duration +5.03% +4.92% +3.48% +3.09% +2.25% +0.91% +1.37% +3.01%
Input token throughput +5.30% +5.15% +3.59% +3.18% +2.31% +0.92% +1.39% +3.12%
Output token throughput +5.29% +5.15% +3.59% +3.18% +2.31% +0.92% +1.39% +3.12%
Total token throughput +5.30% +5.15% +3.59% +3.18% +2.31% +0.92% +1.39% +3.12%
Mean E2E latency +5.04% +4.91% +3.54% +3.09% +2.27% +0.55% +1.19% +2.94%
Median E2E latency +4.64% +4.47% +3.57% +2.78% +2.07% +0.94% +1.00% +2.78%
Mean TTFT +24.47% +19.82% +0.59% +9.13% +6.76% -4.46% +5.92% +8.89%
Median TTFT -2.09% -2.86% +4.38% -3.61% +2.38% +5.67% +7.71% +1.65%
Mean TPOT +4.58% +4.70% +3.76% +2.81% +2.05% +0.80% +0.77% +2.78%
Median TPOT +4.58% +4.38% +3.75% +2.84% +1.90% +0.79% +0.64% +2.70%
Mean ITL +4.58% +4.70% +3.76% +2.81% +2.05% +0.80% +0.77% +2.78%
Median ITL +4.61% +4.73% +3.47% +3.11% +1.93% +0.76% +0.43% +2.72%

Modifications

Fuse RMSNorm with downstream static per-tensor FP8 activation quantization using FlashInfer's rmsnorm_quant and fused_add_rmsnorm_quant kernels. The norm can emit an already-quantized FP8 activation, allowing the following FP8 linear layer to skip its activation quantization.

On SM90, channelwise CUTLASS FP8 GEMMs now consume the scalar activation scale directly instead of expanding it per token, eliminating the extra copy that reduced the fusion benefit.

Changes

  • Add fused RMSNorm + static FP8 quantization, including residual paths and FP16/BF16 output dtype propagation.
  • Enable compatible FP8 linear layers to consume pre-quantized (fp8, scale, original_dtype) inputs.
  • Support both native FP8 and compressed-tensors W8A8 FP8 linear methods.
  • Pass downstream projections to Llama and Qwen2 norms so the fused path can be selected.
  • Add --enable-flashinfer-rmsnorm-fp8-quant, disabled by default.
  • Add native scalar-A/channelwise-B broadcasting to the SM90,SM100,SM120 CUTLASS FP8 GEMM while preserving dynamic per-token and non-SM90,SM100,SM120 behavior. These changes give a small boost to without fusions also
  • Add correctness, dispatch, dtype, scalar-broadcast, and benchmark coverage.

In future PR's I hope to expand model support for this fusion

Accuracy Tests

GSM8k H100

RedHatAI/Meta-Llama-3.1-8B-FP8

# Default
Accuracy: 0.514
Invalid: 0.002
Latency: 12.878 s
Output throughput: 9500.232 token/s

# Fusion
Accuracy: 0.509
Invalid: 0.002
Latency: 12.595 s
Output throughput: 9781.766 token/s

RedHatAI/Llama-2-7b-chat-hf-FP8

# Default
Accuracy: 0.255
Invalid: 0.005
Latency: 36.865 s
Output throughput: 3994.299 token/s

# Fusion
Accuracy: 0.256
Invalid: 0.005
Latency: 36.505 s
Output throughput: 4061.120 token/s

RedHatAI/Qwen2-7B-Instruct-FP8

# Default
Accuracy: 0.765
Invalid: 0.008
Latency: 13.055 s
Output throughput: 13618.943 token/s

# Fusion
Accuracy: 0.765
Invalid: 0.006
Latency: 12.855 s
Output throughput: 13754.316 token/s

Speed Tests and Profiling

Kernel Benchmark H100

➜ python3 benchmark/kernels/bench_fused_rmsnorm_fp8_quant.py
correctness check passed (all fused providers vs unfused within FP8)
rmsnorm_fp8_quant_h4096_residualFalse:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.007168                                       0.003779                                             0.004515
1      1024.0                                             0.011040                                       0.008006                                             0.008467
2      2048.0                                             0.022272                                       0.013434                                             0.014083
3      4096.0                                             0.045466                                       0.027411                                             0.026645
4      8192.0                                             0.110162                                       0.049283                                             0.049408
5     16384.0                                             0.217082                                       0.094088                                             0.094354
rmsnorm_fp8_quant_h4096_residualTrue:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.011722                                       0.009744                                             0.010122
1      1024.0                                             0.018976                                       0.015466                                             0.015904
2      2048.0                                             0.039446                                       0.028643                                             0.028883
3      4096.0                                             0.083677                                       0.057682                                             0.058045
4      8192.0                                             0.173251                                       0.110624                                             0.110846
5     16384.0                                             0.340301                                       0.215798                                             0.216066
rmsnorm_fp8_quant_h8192_residualFalse:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.010845                                       0.007914                                             0.008381
1      1024.0                                             0.019192                                       0.013232                                             0.014144
2      2048.0                                             0.044971                                       0.027181                                             0.026547
3      4096.0                                             0.109398                                       0.049373                                             0.049333
4      8192.0                                             0.216126                                       0.094400                                             0.094349
5     16384.0                                             0.426589                                       0.183293                                             0.182787
rmsnorm_fp8_quant_h8192_residualTrue:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.018870                                       0.015427                                             0.015987
1      1024.0                                             0.039370                                       0.028570                                             0.029037
2      2048.0                                             0.083394                                       0.057946                                             0.058448
3      4096.0                                             0.173722                                       0.110605                                             0.111098
4      8192.0                                             0.340426                                       0.215459                                             0.216043
5     16384.0                                             0.673446                                       0.426339                                             0.426995

Kernel Benchmark B200

correctness check passed (all fused providers vs unfused within FP8)
rmsnorm_fp8_quant_h4096_residualFalse:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.004240                                       0.002806                                             0.003213
1      1024.0                                             0.006502                                       0.003814                                             0.004237
2      2048.0                                             0.010176                                       0.006694                                             0.007504
3      4096.0                                             0.019589                                       0.011798                                             0.011808
4      8192.0                                             0.042762                                       0.023654                                             0.022640
5     16384.0                                             0.079254                                       0.039635                                             0.041277
rmsnorm_fp8_quant_h4096_residualTrue:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.005882                                       0.003626                                             0.004035
1      1024.0                                             0.009562                                       0.007309                                             0.007923
2      2048.0                                             0.016736                                       0.012429                                             0.013040
3      4096.0                                             0.031651                                       0.023152                                             0.023456
4      8192.0                                             0.062982                                       0.044850                                             0.044755
5     16384.0                                             0.132595                                       0.089194                                             0.088966
rmsnorm_fp8_quant_h8192_residualFalse:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.007472                                       0.003626                                             0.004237
1      1024.0                                             0.010362                                       0.006685                                             0.007718
2      2048.0                                             0.018986                                       0.012208                                             0.012003
3      4096.0                                             0.036384                                       0.024069                                             0.023456
4      8192.0                                             0.090984                                       0.045366                                             0.044974
5     16384.0                                             0.176336                                       0.088685                                             0.089002
rmsnorm_fp8_quant_h8192_residualTrue:
   num_tokens  rmsnorm + static_quant_fp8 (unfused) (latency (ms))  rmsnorm_quant (fused, default) (latency (ms))  rmsnorm_quant_cute (fused, cute-dsl) (latency (ms))
0       512.0                                             0.010154                                       0.007312                                             0.007923
1      1024.0                                             0.016941                                       0.012650                                             0.013232
2      2048.0                                             0.030954                                       0.023459                                             0.023472
3      4096.0                                             0.066266                                       0.044230                                             0.044931
4      8192.0                                             0.133942                                       0.089082                                             0.089421
5     16384.0                                             0.272381                                       0.180227                                             0.179595

Bench one batch H100

RedHatAI/Meta-Llama-3.1-8B-FP8

# Default
python3 -m sglang.benchmark.one_batch --model RedHatAI/Meta-Llama-3.1-8B-FP8 --batch-size 1 --input-len 1024 --output-len 1024
Prefill. latency: 0.01953 s, throughput:  52423.30 token/s
Decode.  median latency: 0.00619 s, median throughput:    161.50 token/s
Total. latency:  6.367 s, throughput:    321.66 token/s

# Fusion
python3 -m sglang.benchmark.one_batch --model RedHatAI/Meta-Llama-3.1-8B-FP8 --batch-size 1 --input-len 1024 --output-len 1024 --enable-flashinfer-rmsnorm-fp8-quant
Prefill. latency: 0.01911 s, throughput:  53577.51 token/s
Decode.  median latency: 0.00603 s, median throughput:    165.81 token/s
Total. latency:  6.195 s, throughput:    330.60 token/s

RedHatAI/Llama-2-7b-chat-hf-FP8

# Default
python3 -m sglang.benchmark.one_batch --model RedHatAI/Llama-2-7b-chat-hf-FP8 --batch-size 1 --input-len 1024 --output-len 1024
Prefill. latency: 0.01761 s, throughput:  58135.32 token/s
Decode.  median latency: 0.00564 s, median throughput:    177.22 token/s
Total. latency:  5.806 s, throughput:    352.75 token/s

# Fusion
python3 -m sglang.benchmark.one_batch --model RedHatAI/Llama-2-7b-chat-hf-FP8 --batch-size 1 --input-len 1024 --output-len 1024 --enable-flashinfer-rmsnorm-fp8-quant
Prefill. latency: 0.01716 s, throughput:  59669.24 token/s
Decode.  median latency: 0.00553 s, median throughput:    180.95 token/s
Total. latency:  5.682 s, throughput:    360.42 token/s

RedHatAI/Qwen2-7B-Instruct-FP8

# Default
python3 -m sglang.benchmark.one_batch --model RedHatAI/Qwen2-7B-Instruct-FP8 --batch-size 1 --input-len 1024 --output-len 1024
Prefill. latency: 0.01868 s, throughput:  54827.40 token/s
Decode.  median latency: 0.00554 s, median throughput:    180.40 token/s
Total. latency:  5.708 s, throughput:    358.82 token/s

# Fusion
python3 -m sglang.benchmark.one_batch --model RedHatAI/Qwen2-7B-Instruct-FP8 --batch-size 1 --input-len 1024 --output-len 1024 --enable-flashinfer-rmsnorm-fp8-quant
Prefill. latency: 0.01828 s, throughput:  56016.34 token/s
Decode.  median latency: 0.00545 s, median throughput:    183.40 token/s
Total. latency:  5.612 s, throughput:    364.93 token/s

Bench One Batch B200

RedHatAI/Meta-Llama-3.1-8B-FP8

# Default
python3 -m sglang.benchmark.one_batch --model RedHatAI/Meta-Llama-3.1-8B-FP8 --batch-size 1 --input-len 1024 --output-len 1024
Prefill. latency: 0.01092 s, throughput:  93757.65 token/s
Decode.  median latency: 0.00317 s, median throughput:    315.30 token/s
Total. latency:  3.271 s, throughput:    626.19 token/s

# Fusion
python3 -m sglang.benchmark.one_batch --model RedHatAI/Meta-Llama-3.1-8B-FP8 --batch-size 1 --input-len 1024 --output-len 1024 --enable-flashinfer-rmsnorm-fp8-quant
Prefill. latency: 0.01081 s, throughput:  94708.97 token/s
Decode.  median latency: 0.00303 s, median throughput:    330.34 token/s
Total. latency:  3.112 s, throughput:    658.17 token/s

RedHatAI/Llama-2-7b-chat-hf-FP8

# Default
Prefill. latency: 0.01110 s, throughput:  92271.37 token/s
Decode.  median latency: 0.00312 s, median throughput:    320.31 token/s
Total. latency:  3.206 s, throughput:    638.86 token/s

# Fusion
python3 -m sglang.benchmark.one_batch --model RedHatAI/Llama-2-7b-chat-hf-FP8 --batch-size 1 --input-len 1024 --output-len 1024 --enable-flashinfer-rmsnorm-fp8-quant
Prefill. latency: 0.01095 s, throughput:  93514.13 token/s
Decode.  median latency: 0.00305 s, median throughput:    328.17 token/s
Total. latency:  3.130 s, throughput:    654.25 token/s

RedHatAI/Qwen2-7B-Instruct-FP8

# Default
python3 -m sglang.benchmark.one_batch --model RedHatAI/Qwen2-7B-Instruct-FP8 --batch-size 1 --input-len 1024 --output-len 1024
Prefill. latency: 0.01124 s, throughput:  91125.73 token/s
Decode.  median latency: 0.00308 s, median throughput:    324.19 token/s
Total. latency:  3.170 s, throughput:    646.12 token/s

# Fusion
python3 -m sglang.benchmark.one_batch --model RedHatAI/Qwen2-7B-Instruct-FP8 --batch-size 1 --input-len 1024 --output-len 1024 --enable-flashinfer-rmsnorm-fp8-quant
Prefill. latency: 0.01130 s, throughput:  90636.21 token/s
Decode.  median latency: 0.00294 s, median throughput:    339.75 token/s
Total. latency:  3.027 s, throughput:    676.68 token/s

Online benchmark sweep H100

RedHatAI/Meta-Llama-3.1-8B-FP8

# Default
================================================== Multi-Benchmark Summary (online) ==================================================
config 0: num_prompts=40 rate=inf max_concurrency=4 isl=1024 osl=1024
config 1: num_prompts=80 rate=inf max_concurrency=8 isl=1024 osl=1024
config 2: num_prompts=160 rate=inf max_concurrency=16 isl=1024 osl=1024
config 3: num_prompts=320 rate=inf max_concurrency=32 isl=1024 osl=1024
config 4: num_prompts=640 rate=inf max_concurrency=64 isl=1024 osl=1024
config 5: num_prompts=1280 rate=inf max_concurrency=128 isl=1024 osl=1024
config 6: num_prompts=2560 rate=inf max_concurrency=256 isl=1024 osl=1024
--------------------------------------------------------------------------------------------------------------------------------------
config                                           0             1             2             3             4             5             6
--------------------------------------------------------------------------------------------------------------------------------------
Benchmark duration (s)                       64.72         69.27         77.88         96.07        136.45        214.77        368.95
Input token throughput (tok/s)              632.23       1181.47       2101.82       3407.39       4798.40       6096.87       7098.11
Output token throughput (tok/s)             632.84       1182.62       2103.88       3410.72       4803.09       6102.83       7105.05
Peak output token throughput (tok/s)        660.00       1242.00       2288.00       3935.00       6016.00       8448.00      15200.00
Total token throughput (tok/s)             1265.07       2364.09       4205.70       6818.12       9601.49      12199.71      14203.16
Mean E2E Latency (ms)                      6470.33       6924.39       7783.27       9575.65      13632.69      21430.20      36809.18
Median E2E Latency (ms)                    6471.11       6926.92       7782.33       9653.67      13678.00      21406.72      36745.68
Mean TTFT (ms)                               61.13         73.61        129.02        194.01        628.32       1318.34       2411.60
Median TTFT (ms)                             63.37         37.98         64.29        127.25        611.79       1333.52       2395.03
Mean TPOT (ms)                                6.27          6.70          7.48          9.17         12.71         19.66         33.62
Median TPOT (ms)                              6.26          6.68          7.45          9.20         12.70         19.67         33.66
Mean ITL (ms)                                 6.27          6.70          7.48          9.17         12.71         19.66         33.62
Median ITL (ms)                               6.25          6.67          7.42          8.99         12.24         18.45         30.50
======================================================================================================================================

# Fused
================================================== Multi-Benchmark Summary (online) ==================================================
config 0: num_prompts=40 rate=inf max_concurrency=4 isl=1024 osl=1024
config 1: num_prompts=80 rate=inf max_concurrency=8 isl=1024 osl=1024
config 2: num_prompts=160 rate=inf max_concurrency=16 isl=1024 osl=1024
config 3: num_prompts=320 rate=inf max_concurrency=32 isl=1024 osl=1024
config 4: num_prompts=640 rate=inf max_concurrency=64 isl=1024 osl=1024
config 5: num_prompts=1280 rate=inf max_concurrency=128 isl=1024 osl=1024
config 6: num_prompts=2560 rate=inf max_concurrency=256 isl=1024 osl=1024
--------------------------------------------------------------------------------------------------------------------------------------
config                                           0             1             2             3             4             5             6
--------------------------------------------------------------------------------------------------------------------------------------
Benchmark duration (s)                       63.60         68.15         76.59         94.95        134.84        213.29        365.83
Input token throughput (tok/s)              643.44       1200.94       2137.15       3447.76       4855.42       6139.23       7158.81
Output token throughput (tok/s)             644.07       1202.12       2139.24       3451.13       4860.16       6145.23       7165.81
Peak output token throughput (tok/s)        664.00       1256.00       2304.00       3936.00       6080.00       9344.00      14080.00
Total token throughput (tok/s)             1287.51       2403.06       4276.38       6898.90       9715.58      12284.46      14324.62
Mean E2E Latency (ms)                      6357.53       6811.92       7654.75       9472.34      13471.10      21229.56      36498.12
Median E2E Latency (ms)                    6357.49       6814.04       7655.94       9604.11      13470.32      21202.18      36477.78
Mean TTFT (ms)                               60.40         74.42        127.52        224.22        607.20       1121.80       2375.07
Median TTFT (ms)                             61.49         44.12         64.46        125.08        624.05       1084.55       2333.99
Mean TPOT (ms)                                6.16          6.59          7.36          9.04         12.57         19.66         33.36
Median TPOT (ms)                              6.15          6.57          7.32          8.99         12.56         19.65         33.39
Mean ITL (ms)                                 6.16          6.59          7.36          9.04         12.57         19.66         33.36
Median ITL (ms)                               6.13          6.56          7.29          8.85         12.09         18.35         30.16
======================================================================================================================================

Online Benchmark Sweep B200

RedHatAI/Meta-Llama-3.1-8B-FP8

# Default
================================================== Multi-Benchmark Summary (online) ==================================================
config 0: num_prompts=40 rate=inf max_concurrency=4 isl=1024 osl=1024
config 1: num_prompts=80 rate=inf max_concurrency=8 isl=1024 osl=1024
config 2: num_prompts=160 rate=inf max_concurrency=16 isl=1024 osl=1024
config 3: num_prompts=320 rate=inf max_concurrency=32 isl=1024 osl=1024
config 4: num_prompts=640 rate=inf max_concurrency=64 isl=1024 osl=1024
config 5: num_prompts=1280 rate=inf max_concurrency=128 isl=1024 osl=1024
config 6: num_prompts=2560 rate=inf max_concurrency=256 isl=1024 osl=1024
--------------------------------------------------------------------------------------------------------------------------------------
config                                           0             1             2             3             4             5             6
--------------------------------------------------------------------------------------------------------------------------------------
Benchmark duration (s)                       29.60         31.11         34.22         41.81         57.72         96.27        174.72
Input token throughput (tok/s)             1382.27       2630.84       4783.11       7830.33      11343.08      13601.41      14989.35
Output token throughput (tok/s)            1383.63       2633.42       4787.78       7837.99      11354.17      13614.70      15004.00
Peak output token throughput (tok/s)       1424.00       2744.00       5238.00       8864.00      13504.00      22784.00      36958.00
Total token throughput (tok/s)             2765.90       5264.26       9570.89      15668.32      22697.26      27216.11      29993.36
Mean E2E Latency (ms)                      2958.46       3108.42       3391.68       4174.16       5760.80       9543.54      17368.36
Median E2E Latency (ms)                    2946.54       3092.81       3362.01       4162.93       5729.51       9520.00      17144.31
Mean TTFT (ms)                               50.48         58.83        130.07        160.04        271.62        567.52       1426.95
Median TTFT (ms)                             41.07         36.36         53.84         85.00        147.36        521.84       1405.83
Mean TPOT (ms)                                2.84          2.98          3.19          3.92          5.37          8.77         15.58
Median TPOT (ms)                              2.84          2.97          3.20          3.88          5.26          8.82         15.60
Mean ITL (ms)                                 2.84          2.98          3.19          3.92          5.37          8.77         15.58
Median ITL (ms)                               2.82          2.96          3.17          3.86          5.19          7.91         11.54
======================================================================================================================================

# Fusion
================================================== Multi-Benchmark Summary (online) ==================================================
config 0: num_prompts=40 rate=inf max_concurrency=4 isl=1024 osl=1024
config 1: num_prompts=80 rate=inf max_concurrency=8 isl=1024 osl=1024
config 2: num_prompts=160 rate=inf max_concurrency=16 isl=1024 osl=1024
config 3: num_prompts=320 rate=inf max_concurrency=32 isl=1024 osl=1024
config 4: num_prompts=640 rate=inf max_concurrency=64 isl=1024 osl=1024
config 5: num_prompts=1280 rate=inf max_concurrency=128 isl=1024 osl=1024
config 6: num_prompts=2560 rate=inf max_concurrency=256 isl=1024 osl=1024
--------------------------------------------------------------------------------------------------------------------------------------
config                                           0             1             2             3             4             5             6
--------------------------------------------------------------------------------------------------------------------------------------
Benchmark duration (s)                       28.11         29.58         33.03         40.52         56.42         95.39        172.33
Input token throughput (tok/s)             1455.47       2766.35       4954.93       8079.14      11604.76      13726.93      15197.19
Output token throughput (tok/s)            1456.89       2769.06       4959.77       8087.03      11616.10      13740.35      15212.04
Peak output token throughput (tok/s)       1496.00       2888.00       5392.00       9192.00      14080.00      20464.00      37596.00
Total token throughput (tok/s)             2912.36       5535.41       9914.70      16166.17      23220.86      27467.28      30409.23
Mean E2E Latency (ms)                      2809.48       2955.87       3271.49       4045.20       5629.94       9491.02      17161.03
Median E2E Latency (ms)                    2809.78       2954.48       3241.87       4047.18       5611.05       9430.20      16973.67
Mean TTFT (ms)                               38.13         47.17        129.30        145.43        253.26        592.82       1342.45
Median TTFT (ms)                             41.93         37.40         51.48         88.07        143.85        492.25       1297.43
Mean TPOT (ms)                                2.71          2.84          3.07          3.81          5.26          8.70         15.46
Median TPOT (ms)                              2.71          2.84          3.08          3.77          5.16          8.75         15.50
Mean ITL (ms)                                 2.71          2.84          3.07          3.81          5.26          8.70         15.46
Median ITL (ms)                               2.69          2.82          3.06          3.74          5.09          7.85         11.49
======================================================================================================================================

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #30863863022
Latest PR Test (Extra): ⏳ Run #30863862825

@gemini-code-assist

gemini-code-assist Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added the quant LLM Quantization label Jul 31, 2026
@DevashishLal-CB
DevashishLal-CB force-pushed the dev/dlal/norm-quant-fusion branch 2 times, most recently from 2259dc5 to 2694840 Compare July 31, 2026 04:28
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 31, 2026
@DevashishLal-CB DevashishLal-CB changed the title feat: add flashinfer rmsnorm + quant fusion support add flashinfer rmsnorm + quant fusion support Jul 31, 2026
@DevashishLal-CB
DevashishLal-CB force-pushed the dev/dlal/norm-quant-fusion branch 2 times, most recently from b1c70aa to 700395c Compare July 31, 2026 07:32
@DevashishLal-CB
DevashishLal-CB marked this pull request as ready for review July 31, 2026 07:45
@gemini-code-assist

gemini-code-assist Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@DevashishLal-CB

DevashishLal-CB commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@DevashishLal-CB

DevashishLal-CB commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@DevashishLal-CB

DevashishLal-CB commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@DevashishLal-CB

DevashishLal-CB commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@DevashishLal-CB

DevashishLal-CB commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-stage base-b-test-1-gpu-small

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

⚠️ /rerun-stage has been deprecated.

Stage granularity is too coarse — a stage usually doesn't map to one feature, so rerunning a stage re-pays the cost of unrelated tests. If you don't know which exact test files to rerun, you shouldn't be using /rerun-stage or /rerun-test in the first place.

Use one of these instead:

  • Selective tests (you know exactly which files to rerun): 
    /rerun-test test_foo.py test_bar.py
  • Rerun only failed jobs: 
    /rerun-failed-ci
  • Full CI rerun (with extra coverage): add the run-ci or run-ci-extra label and push a new commit (or use /tag-and-rerun-ci).

AMD CI: stage-level dispatch is still available via Actions UI → PR Test (AMD) / PR Test ROCm 7.2 (AMD) → Run workflow → pick a stage from the dropdown.

@DevashishLal-CB

DevashishLal-CB commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

#33441, hopefully this helps with dependency failures

@BBuf
BBuf merged commit 3960983 into sgl-project:main Aug 4, 2026
99 of 187 checks passed
@Fridge003 Fridge003 mentioned this pull request Aug 4, 2026
Merged
@hnyls2002 hnyls2002 mentioned this pull request Aug 4, 2026
Closed
This was referenced Aug 4, 2026
Merged
Merged
DevashishLal-CB pushed a commit to DevashishLal-CB/sglang that referenced this pull request Aug 6, 2026
Devashish Lal
runtime changes for sgl-project#32994
ccf5aff 
Signed-off-by: Devashish Lal <devcode@fb.com>
DevashishLal-CB pushed a commit to DevashishLal-CB/sglang that referenced this pull request Aug 6, 2026
Devashish Lal
runtime changes for sgl-project#32994
7c79e35 
Signed-off-by: Devashish Lal <devcode@fb.com>
kfhfar pushed a commit to kfhfar/sglang that referenced this pull request Aug 7, 2026
@DevashishLal-CB Devashish Lal
@BBuf @kfhfar
Add flashinfer rmsnorm + quant fusion support SM90, SM100, SM120 (sgl…
c788530 
…-project#32994)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
BBuf added a commit that referenced this pull request Aug 9, 2026
@DevashishLal-CB Devashish Lal @BBuf
runtime: Add flashinfer rmsnorm + quant fusion support SM90, SM100, S…
11d03ea 
…M120- #32994 (#33471)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Xia-Weiwen pushed a commit to Xia-Weiwen/sglang that referenced this pull request Aug 10, 2026
@DevashishLal-CB Devashish Lal
@BBuf @Xia-Weiwen
runtime: Add flashinfer rmsnorm + quant fusion support SM90, SM100, S…
461441b 
…M120- sgl-project#32994 (sgl-project#33471)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 16, 2026
@DevashishLal-CB Devashish Lal
@BBuf @saturn-acc
Add flashinfer rmsnorm + quant fusion support SM90, SM100, SM120 (sgl…
a64770e 
…-project#32994)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 16, 2026
@DevashishLal-CB Devashish Lal
@BBuf @saturn-acc
runtime: Add flashinfer rmsnorm + quant fusion support SM90, SM100, S…
fa6521b 
…M120- sgl-project#32994 (sgl-project#33471)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
@DevashishLal-CB Devashish Lal
@BBuf @jakki-amd
Add flashinfer rmsnorm + quant fusion support SM90, SM100, SM120 (sgl…
be79f40 
…-project#32994)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
@DevashishLal-CB Devashish Lal
@BBuf @jakki-amd
runtime: Add flashinfer rmsnorm + quant fusion support SM90, SM100, S…
6465ba8 
…M120- sgl-project#32994 (sgl-project#33471)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
@DevashishLal-CB Devashish Lal @BBuf
Add flashinfer rmsnorm + quant fusion support SM90, SM100, SM120 (sgl…
8e07323 
…-project#32994)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
@DevashishLal-CB Devashish Lal @BBuf
runtime: Add flashinfer rmsnorm + quant fusion support SM90, SM100, S…
d087477 
…M120- sgl-project#32994 (sgl-project#33471)

Signed-off-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Devashish Lal <devcode@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BBuf BBuf BBuf approved these changes
@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner
@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner
@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner
@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner
@AniZpZ AniZpZ Awaiting requested review from AniZpZ AniZpZ is a code owner
@b8zhong b8zhong Awaiting requested review from b8zhong b8zhong is a code owner
@OrangeRedeng OrangeRedeng Awaiting requested review from OrangeRedeng OrangeRedeng is a code owner
@Alisehen Alisehen Awaiting requested review from Alisehen Alisehen is a code owner
@mmangkad mmangkad Awaiting requested review from mmangkad mmangkad is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner
@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner
@wisclmy0611 wisclmy0611 Awaiting requested review from wisclmy0611
@zijiexia zijiexia Awaiting requested review from zijiexia
@JustinTong0323 JustinTong0323 Awaiting requested review from JustinTong0323
@sogalin sogalin Awaiting requested review from sogalin

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation flashinfer high priority performance quant LLM Quantization run-ci run-ci-extra sgl-kernel

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DevashishLal-CB @BBuf