[Bugfix][Misc] Fix silu_and_mul_nvfp4_quant issue and extract common utils for nvfp4 kernel source files by elvischenv · Pull Request #23727 · vllm-project/vllm

elvischenv · 2025-08-27T09:37:34Z

Purpose

Fixed the undefined symbol issue by keeping the function definition even on non-nvfp4-supported platforms in csrc/quantization/fp4/nvfp4_quant_entry.cu
Extracted common utils for nvfp4 kernel source files.
Removed unnecessary CUDA_ARCH checking for the nvfp4 kernel.
Switched to use VLLM_DISPATCH_HALF_TYPES for the nvfp4_quant_kernels.cu and nvfp4_experts_quant.cu

Test Plan && Test Result

tests/kernels/quantization/test_silu_nvfp4_quant_fusion.py
==== 8 passed in 2.49s ===
tests/kernels/quantization/test_nvfp4_quant.py
====== 18 passed in 5.58s =========
tests/kernels/moe/test_nvfp4_moe.py
======= 180 passed in 30.85s ======

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

gemini-code-assist

Code Review

This pull request refactors common CUDA kernel utilities for nvfp4 quantization into a new header file, csrc/quantization/fp4/nvfp4_utils.cuh. This is a good change that improves code reuse and maintainability. My review focuses on the new utility file, and I've identified a few areas for improvement to make the utilities safer for future use. Specifically, several functions designed for newer hardware architectures (SM100+) have fallback paths for older architectures that fail silently by returning 0 or nullptr. This could lead to hard-to-debug issues if these utilities are reused without the proper architecture guards. I've suggested using static_assert to cause a compile-time error in these cases, making any misuse immediately obvious.

csrc/quantization/fp4/nvfp4_utils.cuh

elvischenv · 2025-08-27T09:45:45Z

csrc/quantization/fp4/nvfp4_utils.cuh

@pavanimajety @ProExpertProg is asking for removing these cuda sm100 macros for these nvfp4 kernels. Do you think this is safe to remove? Thanks

If we want to be conservative we can add an #error macro if the CUDA arch is too low

I think it should be safe. Removed them.

Sorry for the delayed reply. As long as these files are not built for lower unsupported architectures, we should be okay to remove it. We may also have to revisit these compilations for SM120+ builds

mergify · 2025-08-29T16:40:21Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @elvischenv.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

ProExpertProg

Looks good just left a static_cast nit!

csrc/quantization/fp4/nvfp4_quant_kernels.cu

ProExpertProg · 2025-09-02T19:05:57Z

Can you make sure to also revert #23959?

ProExpertProg

Another reinterpret_cast, will just fix directly

csrc/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu

MatthewBonanni · 2025-09-02T21:30:36Z

I made #24121 as a workaround, if this lands first I'll close it!

MatthewBonanni · 2025-09-03T00:42:48Z

#24121 has been merged, so please revert those before merging this PR. Thanks!

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

This reverts commit e66ed3e. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

…ject#24121)" This reverts commit 2fd1a40. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

This reverts commit 41be13dcbb179cc6fa78bf176069858dbe6087e9. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com> Revert "fix pre-commit" This reverts commit ebac9cd0f6455a9f3580454a0b48ebaf95b1f4bc. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

…utils for nvfp4 kernel source files (vllm-project#23727) Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com> Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

…ithout nvfp4 Fixes the undefined symbol issue (vllm-project#23925 (comment)) by keeping the function definition of `cutlass_fp4_group_mm` even on non-nvfp4-supported platforms in `csrc/quantization/fp4/nvfp4_blockwise_moe_entry.cu`. Similar to vllm-project#23727.

gemini-code-assist bot reviewed Aug 27, 2025

View reviewed changes

elvischenv commented Aug 27, 2025

View reviewed changes

elvischenv force-pushed the elvischenv/nvfp4-utils branch 3 times, most recently from 7465d77 to 29dbc72 Compare August 29, 2025 16:35

elvischenv requested review from youkaichao and zou3519 as code owners August 29, 2025 16:35

mergify bot added the needs-rebase label Aug 29, 2025

zou3519 mentioned this pull request Aug 29, 2025

[Bug]: _C.abi3.so: undefined symbol: _Z24silu_and_mul_nvfp4_quantRN2at6TensorES1_S1_S1_ #23925

Closed

1 task

elvischenv force-pushed the elvischenv/nvfp4-utils branch from 29dbc72 to 7794ce5 Compare August 29, 2025 16:48

mergify bot removed the needs-rebase label Aug 29, 2025

ProExpertProg approved these changes Sep 2, 2025

View reviewed changes

csrc/quantization/fp4/nvfp4_quant_kernels.cu Outdated Show resolved Hide resolved

elvischenv force-pushed the elvischenv/nvfp4-utils branch from 7794ce5 to 7eca643 Compare September 2, 2025 14:12

ProExpertProg approved these changes Sep 2, 2025

View reviewed changes

csrc/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu Outdated Show resolved Hide resolved

elvischenv requested review from WoosukKwon, tlrmchlsmth and yewentao256 as code owners September 2, 2025 23:09

elvischenv force-pushed the elvischenv/nvfp4-utils branch from 2cb160e to e897656 Compare September 2, 2025 23:11

MatthewBonanni mentioned this pull request Sep 3, 2025

[CI/Build] Disable SiluMul NVFP4 quant fusion tests #24121

Merged

5 tasks

elvischenv force-pushed the elvischenv/nvfp4-utils branch from e897656 to e52bbe9 Compare September 3, 2025 00:48

mergify bot added the ci/build label Sep 3, 2025

elvischenv force-pushed the elvischenv/nvfp4-utils branch 2 times, most recently from 7979552 to ec829a6 Compare September 3, 2025 00:53

elvischenv changed the title ~~[Misc] Extract common utils for nvfp4 kernel source files~~ [Bugfix][Misc] Fix silu_and_mul_nvfp4_quant issue and extract common utils for nvfp4 kernel source files Sep 3, 2025

elvischenv added 2 commits September 3, 2025 05:33

move common nvfp4 utils to nvfp4_utils.cuh

15831a5

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

remove sm100 arch macro for nvfp4 kernel

a826562

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

elvischenv and others added 7 commits September 3, 2025 05:33

revert part of vllm-project#23929

6190d91

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

reinterpret_cast to static_cast

2913045

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

static cast

4f61c56

Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

fix pre-commit

aca3a14

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

Revert "[CI Failure] Skip failing nvfp4 silu test (vllm-project#23959)"

8a1b505

This reverts commit e66ed3e. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

Revert "[CI/Build] Disable SiluMul NVFP4 quant fusion tests (vllm-pro…

c66bd6a

…ject#24121)" This reverts commit 2fd1a40. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

elvischenv force-pushed the elvischenv/nvfp4-utils branch from ec829a6 to ed01518 Compare September 3, 2025 12:33

ProExpertProg approved these changes Sep 3, 2025

View reviewed changes

ProExpertProg enabled auto-merge (squash) September 3, 2025 15:07

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 3, 2025

Merge branch 'main' into elvischenv/nvfp4-utils

c80de93

Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>

simon-mo disabled auto-merge September 4, 2025 21:25

simon-mo merged commit adc3ddb into vllm-project:main Sep 4, 2025
68 of 71 checks passed

elvischenv deleted the elvischenv/nvfp4-utils branch September 5, 2025 01:35

StrongerXi mentioned this pull request Oct 2, 2025

[Bugfix] Fix build issue around cutlass_fp4_group_mm on platforms without nvfp4 #26061

Closed

5 tasks

ProExpertProg linked an issue Jan 2, 2026 that may be closed by this pull request

[Bug]: _C.abi3.so: undefined symbol: _Z24silu_and_mul_nvfp4_quantRN2at6TensorES1_S1_S1_ #23925

Closed

1 task

Uh oh!

Conversation

elvischenv commented Aug 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan && Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elvischenv Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ProExpertProg Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

elvischenv Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

pavanimajety Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Aug 29, 2025

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ProExpertProg commented Sep 2, 2025

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MatthewBonanni commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MatthewBonanni commented Sep 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elvischenv commented Aug 27, 2025 •

edited by github-actions bot

Loading

MatthewBonanni commented Sep 2, 2025 •

edited

Loading