Skip to content

Expose flashinfer_cute_dsl as fp8 gemm backend - #34435

Closed
IwakuraRein wants to merge 1 commit into
sgl-project:mainfrom
IwakuraRein:expose-flashinfer-cutedsl
Closed

IwakuraRein wants to merge 1 commit into
sgl-project:mainfrom
IwakuraRein:expose-flashinfer-cutedsl

Conversation

@IwakuraRein

@IwakuraRein IwakuraRein commented Aug 11, 2026

Copy link
Copy Markdown

Motivation

Expose flashinfer_cute_dsl as fp8 gemm backend since it achieves better perf for muse glimmer on B200/300.

PR dependency: #34262.

Modifications

Accuracy Tests

Speed Tests and Profiling

Benchmark Muse Glimmer on a single B300, ISL 2048, OSL 256:

image

Flashinfer benchmark on a single B300 with Muse Glimmer shape

N=6656, K=19968

M cutlass (TFLOPs/s) cute-dsl (TFLOPs/s)
1 6.312 8.555
32 208.153 256.573
64 415.487 494.538
128 817.889 982.648
256 1446.592 1818.284
512 2224.931 2624.488
1024 2708.906 3201.373
2048 2838.883 3499.666
4096 3519.589 3449.986
8192 3403.057 3193.448

N=19968, K=6656

M cutlass (TFLOPs/s) cute-dsl (TFLOPs/s)
1 6.775 7.504
32 224.885 244.536
64 448.619 498.479
128 895.739 1002.112
256 1463.501 1680.373
512 2261.627 2641.604
1024 3029.742 3146.874
2048 3539.739 3382.431
4096 3710.748 3601.933
8192 3635.707 3376.387

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #31618178019
Latest PR Test (Extra): ❌ Run #31618177806

@github-actions github-actions Bot added documentation Improvements or additions to documentation quant LLM Quantization apple-silicon labels Aug 11, 2026
@mmangkad

Copy link
Copy Markdown
Collaborator

Hi @IwakuraRein, this is actually already being worked on in #34042

@IwakuraRein
IwakuraRein force-pushed the expose-flashinfer-cutedsl branch from 5f73816 to fd5f995 Compare August 12, 2026 16:20
…k default

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein
IwakuraRein force-pushed the expose-flashinfer-cutedsl branch from fd5f995 to fbf0d1c Compare August 12, 2026 16:33
@IwakuraRein

Copy link
Copy Markdown
Author

Close due to duplicating #34042

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apple-silicon documentation Improvements or additions to documentation quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants