[elfiegg]

Motivation

We observed ~1.2x e2e perf improvement on Blackwell with CUTLASS for Deepseek V3/R1 when # request = 3000, ISL=OSL=1000.
(The accurate number will be confirmed tonight or tomorrow)
Below is per kernel stats compared to Triton W8A8 on Blackwell. We've noticed for 16 =< batch_size =<128, CUTLASS tends to perform worse than Triton but this will be fixed in the near future.

Sampled DeepSeek-V3 FP8 Blockwise Scaled Matmul Performance

N=24576, K=7168

Batch Size	Triton FP8 Blockwise GEMM	SGL-Kernel FP8 Blockwise GEMM
1.0	6145.284887	4250.963051
16.0	101846.293112	84880.085440
32.0	148584.865407	174810.051297
64.0	293949.025732	343992.068556
128.0	367666.175511	511930.118380
256.0	531211.867847	419627.341862
512.0	591411.059307	431173.185733
1024.0	675422.890263	474651.478739
2048.0	739234.765151	499399.789876

N=32768, K=512

Batch Size	Triton FP8 Blockwise GEMM	SGL-Kernel FP8 Blockwise GEMM
1.0	1272.316929	1067.311184
16.0	21117.608897	37516.459555
32.0	39207.953346	75032.919109
64.0	78415.906692	141903.263079
128.0	162856.566971	249374.113274
256.0	282036.426762	282329.911235
512.0	352591.354020	368389.725161
1024.0	403448.387533	409229.330931
2048.0	429047.692250	432380.936748

N=7168, K=16384

Batch Size	Triton FP8 Blockwise GEMM	SGL-Kernel FP8 Blockwise GEMM
1.0	3953.955728	2696.472995
16.0	51821.761937	55572.343537
32.0	79836.856063	109794.332871
64.0	149990.337657	216055.051678
128.0	220982.714170	431713.122755
256.0	440566.968887	362522.454748
512.0	485328.947619	372342.699456
1024.0	575727.714840	385140.654844
2048.0	687865.688544	443287.829769

N=7168, K=18432

Batch Size	Triton FP8 Blockwise GEMM	SGL-Kernel FP8 Blockwise GEMM
1.0	4093.161548	2867.059998
16.0	52969.941874	56623.819895
32.0	81030.044767	111527.431546
64.0	152918.715055	220542.336211
128.0	224666.384396	440716.945859
256.0	446486.467036	365398.302756
512.0	489368.193526	373399.255231
1024.0	568468.296784	385833.413716
2048.0	638405.936553	443885.604232

Modifications

fp8 blockwise gemm. Dependes on #4277 .

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[elfiegg]

cc @kushanam @depaulmillz

[zhyncs]

@elfiegg Can you help fix the build error and lint CI failure?

[zhyncs]

Hi @elfiegg I will merge this first. Once I have access to the B200, I can help with testing. Thank you for your contribution!