fix cutlass_3x_gemm_fp8_blockwise on sm103a by IwakuraRein · Pull Request #32224 · vllm-project/vllm

IwakuraRein · 2026-01-13T00:45:13Z

Purpose

When compiling with sm103a, the output of cutlass_3x_gemm_fp8_blockwise is garbage value.

Updated the helpers in csrc/cutlass_extensions/common.hpp to include sm103a.

Also added a runtime error message when the kernel is executed but not compiled.

Test Plan

pytest tests/kernels/quantization/test_block_fp8.py::test_w8a8_block_fp8_cutlass_matmul

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

gemini-code-assist

Code Review

This pull request aims to fix an issue with cutlass_3x_gemm_fp8_blockwise on the sm103a architecture by introducing a new preprocessor guard, enable_sm100f_only, which correctly includes sm103a. The change also adds compiler warnings for kernels used on unsupported architectures, which is a good practice. However, the implementation of these warnings is flawed in several places, causing them to be triggered during host code compilation, which can be very noisy. My review includes suggestions to correct this behavior so that warnings are only shown for device code compilation on unsupported architectures.

gemini-code-assist · 2026-01-13T00:46:41Z

  CUTLASS_DEVICE void operator()(Args&&... args) {
 #if defined __CUDA_ARCH__ && __CUDA_ARCH__ >= 900
    Kernel::operator()(std::forward<Args>(args)...);
+#else


The use of #else here will cause a compiler warning to be emitted during host-side compilation, as __CUDA_ARCH__ is not defined. This can lead to a large number of unnecessary warnings. To ensure the warning is only triggered for device compilation on an unsupported architecture, you should use #elif defined(__CUDA_ARCH__). This pattern is already correctly used for enable_sm100f_only in this same file.

#elif defined(__CUDA_ARCH__)

gemini-code-assist · 2026-01-13T00:46:41Z

  CUTLASS_DEVICE void operator()(Args&&... args) {
 #if defined __CUDA_ARCH__ && __CUDA_ARCH__ == 900
    Kernel::operator()(std::forward<Args>(args)...);
+#else


The use of #else here will cause a compiler warning to be emitted during host-side compilation, as __CUDA_ARCH__ is not defined. This can lead to a large number of unnecessary warnings. To ensure the warning is only triggered for device compilation on an unsupported architecture, you should use #elif defined(__CUDA_ARCH__). This pattern is already correctly used for enable_sm100f_only in this same file.

#elif defined(__CUDA_ARCH__)

gemini-code-assist · 2026-01-13T00:46:41Z

  CUTLASS_DEVICE void operator()(Args&&... args) {
 #if defined __CUDA_ARCH__ && __CUDA_ARCH__ == 1000
    Kernel::operator()(std::forward<Args>(args)...);
+#else


The use of #else here will cause a compiler warning to be emitted during host-side compilation, as __CUDA_ARCH__ is not defined. This can lead to a large number of unnecessary warnings. To ensure the warning is only triggered for device compilation on an unsupported architecture, you should use #elif defined(__CUDA_ARCH__). This pattern is already correctly used for enable_sm100f_only in this same file.

#elif defined(__CUDA_ARCH__)

gemini-code-assist · 2026-01-13T00:46:41Z

  CUTLASS_DEVICE void operator()(Args&&... args) {
 #if defined __CUDA_ARCH__ && __CUDA_ARCH__ == 1200
    Kernel::operator()(std::forward<Args>(args)...);
+#else


The use of #else here will cause a compiler warning to be emitted during host-side compilation, as __CUDA_ARCH__ is not defined. This can lead to a large number of unnecessary warnings. To ensure the warning is only triggered for device compilation on an unsupported architecture, you should use #elif defined(__CUDA_ARCH__). This pattern is already correctly used for enable_sm100f_only in this same file.

#elif defined(__CUDA_ARCH__)

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

vadiklyutiy · 2026-01-13T20:54:34Z

    Kernel::operator()(std::forward<Args>(args)...);
+  #else
+    printf("This kernel only supports sm100a.\n");
+    asm("trap;");


Could you check, in case of real fail, will this print appears?

Yes. I have tested and the print appears.

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

vadiklyutiy

looks good

mxz297 · 2026-01-15T17:12:44Z

Thanks for the identifying and fixing this issue!

IwakuraRein · 2026-01-15T19:03:16Z

Looks like the ci/cd failed due to some bugs in the scaled_dequantize at quant_utils.py

mgoin

LGTM!

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com> Signed-off-by: Pai <416932041@qq.com>

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com> Signed-off-by: felix01.yu <felix01.yu@vipshop.com>

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com>

fix cutlass_3x_gemm_fp8_blockwise on sm103a

819835d

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

mergify Bot added the nvidia label Jan 13, 2026

github-project-automation Bot added this to NVIDIA Jan 13, 2026

gemini-code-assist Bot reviewed Jan 13, 2026

View reviewed changes

IwakuraRein added 2 commits January 13, 2026 01:02

upd

5af851a

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

typo

a22effd

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

IwakuraRein force-pushed the fix-cutlass_3x_gemm_fp8_blockwise branch from 4575a3c to 764db01 Compare January 13, 2026 18:21

replace compile time warning with runtime err

42d3ba8

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

IwakuraRein force-pushed the fix-cutlass_3x_gemm_fp8_blockwise branch from 764db01 to 42d3ba8 Compare January 13, 2026 18:26

typo

b859962

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

IwakuraRein force-pushed the fix-cutlass_3x_gemm_fp8_blockwise branch from 64b930e to b859962 Compare January 13, 2026 18:34

IwakuraRein added 2 commits January 13, 2026 18:36

upd

cac4dd7

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

replace runtime error with trap since it is device code

2d197ba

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

vadiklyutiy reviewed Jan 13, 2026

View reviewed changes

apply helper to gemm

d16cf36

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>

IwakuraRein marked this pull request as ready for review January 13, 2026 21:53

IwakuraRein moved this to Ready in NVIDIA Jan 13, 2026

pavanimajety added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 14, 2026

nvpohanh approved these changes Jan 14, 2026

View reviewed changes

vadiklyutiy approved these changes Jan 14, 2026

View reviewed changes

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

ffc5beb

IwakuraRein and others added 4 commits January 15, 2026 13:30

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

6a1dc07

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

fa7ebab

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

2a58489

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

4f04afa

pacoxu mentioned this pull request Jan 27, 2026

[CI Failure]: Blackwell Test failed for cudaErrorDevicesUnavailable #33153

Closed

3 tasks

pavanimajety and others added 2 commits January 27, 2026 10:07

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

3f05257

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

a76e33c

Merge branch 'main' into fix-cutlass_3x_gemm_fp8_blockwise

448c405

mgoin approved these changes Feb 2, 2026

View reviewed changes

vllm-bot merged commit 089cd4f into vllm-project:main Feb 2, 2026
92 of 95 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Feb 2, 2026

mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026

fix cutlass_3x_gemm_fp8_blockwise on sm103a (vllm-project#32224)

487e7b0

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com>

my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026

fix cutlass_3x_gemm_fp8_blockwise on sm103a (vllm-project#32224)

ccb9a42

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com>

0826joyce pushed a commit to 0826joyce/vllm-serving-optimization that referenced this pull request May 19, 2026

fix cutlass_3x_gemm_fp8_blockwise on sm103a (vllm-project#32224)

f3f1e88

Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com>

Navigation Menu

Uh oh!

Uh oh!

fix cutlass_3x_gemm_fp8_blockwise on sm103a - #32224

fix cutlass_3x_gemm_fp8_blockwise on sm103a#32224
vllm-bot merged 16 commits into
vllm-project:mainfrom
IwakuraRein:fix-cutlass_3x_gemm_fp8_blockwise

IwakuraRein commented Jan 13, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Uh oh!

vadiklyutiy Jan 13, 2026

Uh oh!

IwakuraRein Jan 13, 2026

Uh oh!

vadiklyutiy left a comment

Uh oh!

mxz297 commented Jan 15, 2026

Uh oh!

IwakuraRein commented Jan 15, 2026

Uh oh!

mgoin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Uh oh!

Conversation

IwakuraRein commented Jan 13, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

vadiklyutiy Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

IwakuraRein Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

vadiklyutiy left a comment

Choose a reason for hiding this comment

Uh oh!

mxz297 commented Jan 15, 2026

Uh oh!

IwakuraRein commented Jan 15, 2026

Uh oh!

mgoin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

IwakuraRein commented Jan 13, 2026 •

edited by github-actions Bot

Loading