fix benchmark fp8 blockwise group gemm

[yuan-luo]

Motivation

DeepGemm's signature m_grouped_gemm_fp8_fp8_bf16_nt_contiguous has been changed to m_grouped_fp8_gemm_nt_contiguous, so the benchmark bench_fp8_blockwise_group_gemm.py is broken. This PR is to fix it.

Main:

➜  sglang_dev git:(main) ✗ python ./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py

Benchmark: expected_m_per_group=128, n=512, k=7168, num_groups=256
Traceback (most recent call last):
  File "/sgl-workspace/sglang_dev/./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py", line 330, in <module>
    main()
  File "/sgl-workspace/sglang_dev/./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py", line 326, in main
    benchmark_one_shape(shape_args, args.num_warmup, args.num_run)
  File "/sgl-workspace/sglang_dev/./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py", line 280, in benchmark_one_shape
    average_time, m = kernel_func(
  File "/sgl-workspace/sglang_dev/./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py", line 106, in bench_deepgemm
    run_deepgemm()
  File "/sgl-workspace/sglang_dev/./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py", line 100, in run_deepgemm
    deep_gemm.m_grouped_gemm_fp8_fp8_bf16_nt_contiguous(
AttributeError: module 'deep_gemm' has no attribute 'm_grouped_gemm_fp8_fp8_bf16_nt_contiguous'

This PR:

➜  sglang_dev git:(main) ✗ python ./sgl-kernel/benchmark/bench_fp8_blockwise_group_gemm.py

Benchmark: expected_m_per_group=128, n=512, k=7168, num_groups=256
deepgemm: 492.9952144622803 us
cutlass: 210.7424020767212 us

Benchmark: expected_m_per_group=256, n=512, k=7168, num_groups=256
deepgemm: 491.44320487976074 us
cutlass: 318.9055919647217 us

Benchmark: expected_m_per_group=256, n=256, k=7168, num_groups=256
deepgemm: 492.01598167419434 us
cutlass: 183.2095980644226 us

Benchmark: expected_m_per_group=512, n=256, k=7168, num_groups=256
deepgemm: 471.8592166900635 us
cutlass: 320.3999996185303 us

Benchmark: expected_m_per_group=1, n=512, k=7168, num_groups=256
deepgemm: 470.3775882720947 us
cutlass: 175.3119945526123 us

Benchmark: expected_m_per_group=2, n=256, k=7168, num_groups=256
deepgemm: 440.1088237762451 us
cutlass: 105.60640096664429 us

Benchmark: expected_m_per_group=256, n=4096, k=7168, num_groups=32
deepgemm: 475.15201568603516 us
cutlass: 295.9264039993286 us

Benchmark: expected_m_per_group=512, n=4096, k=7168, num_groups=16
deepgemm: 474.9472141265869 us
cutlass: 273.81439208984375 us

Benchmark: expected_m_per_group=4, n=4096, k=7168, num_groups=32
deepgemm: 461.87520027160645 us
cutlass: 164.88640308380127 us

Benchmark: expected_m_per_group=8, n=4096, k=7168, num_groups=16
deepgemm: 452.09598541259766 us
cutlass: 94.44479942321777 us

Benchmark: expected_m_per_group=1024, n=768, k=4096, num_groups=128
deepgemm: 534.3264102935791 us
cutlass: 529.3888092041016 us

Benchmark: expected_m_per_group=1024, n=4096, k=384, num_groups=128
deepgemm: 479.4015884399414 us
cutlass: 755.3919792175293 us

Benchmark: expected_m_per_group=16, n=768, k=4096, num_groups=128
deepgemm: 453.5520076751709 us
cutlass: 87.94559836387634 us

Benchmark: expected_m_per_group=16, n=4096, k=384, num_groups=128
deepgemm: 436.60478591918945 us
cutlass: 90.79999923706055 us

Modifications

Accuracy Test

Benchmark & Profiling

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[gemini-code-assist]

Summary of Changes

Hello @yuan-luo, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical AttributeError in the fp8 blockwise group gemm benchmark script. The issue arose because an underlying function signature in the DeepGemm library changed. By updating the function call in the benchmark, the script is now functional again, allowing for continued performance evaluation of DeepGemm and Cutlass operations.

Highlights

• Benchmark Fix: Updated the bench_fp8_blockwise_group_gemm.py script to reflect a breaking change in the DeepGemm library's API. The function m_grouped_gemm_fp8_fp8_bf16_nt_contiguous was renamed to m_grouped_fp8_gemm_nt_contiguous, causing an AttributeError in the benchmark. This PR updates the function call to resolve the issue.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request fixes a benchmark script that was broken due to a function rename in the deep_gemm library. The change correctly updates the function call, resolving the AttributeError. The fix is straightforward and well-contained. Please verify that the new function provides the expected functionality and performance.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!