Add cutlass support for blackwell fp8 blockwise gemm by wenscarl · Pull Request #14383 · vllm-project/vllm

wenscarl · 2025-03-06T20:37:28Z

This PR adds support for cutlass blackwell blockwise gemm for fp8.

github-actions · 2025-03-06T20:37:37Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

pathorn · 2025-03-06T21:58:43Z

It looks like scaled_mm_blockwise_sm100_fp8.cu and scaled_mm_blockwise_sm100_fp8_dispatch.cuh were committed only as symlinks. When the change is done and ready for review, make sure to double check the actual files are added.

I'm curious to compare it to the blockwise kernels I wrote in the blackwell-rebase-feb20 branch of deepinfra/vllm , since we had to change the scale factor to be fp8

kushanam · 2025-03-10T21:05:32Z

Thank you @wenscarl! Please correct thecutlass_scaled_mm_supports_block_fp8() function in scaled_mm_entry.cu to allow >100 arch as well.

WoosukKwon · 2025-03-25T16:53:17Z

@tylertitsworth Can you please take a look?

tlrmchlsmth

Thanks for the contribution! A couple of comments but looks good overall

csrc/quantization/cutlass_w8a8/scaled_mm_c3x_sm100.cu

csrc/quantization/cutlass_w8a8/c3x/scaled_mm_blockwise_sm100_fp8.cu

csrc/quantization/cutlass_w8a8/c3x/scaled_mm_blockwise_sm100_fp8_dispatch.cuh

mergify · 2025-03-28T16:14:40Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wenscarl.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

csrc/cutlass_extensions/common.hpp

tlrmchlsmth · 2025-04-04T21:10:09Z

Looks good to me now, thank you!

Please merge in latest main to fix the docker-build-image failure

csrc/quantization/cutlass_w8a8/c3x/scaled_mm_blockwise_sm100_fp8_dispatch.cuh

mergify · 2025-04-23T22:59:50Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wenscarl.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

vllm/model_executor/layers/quantization/utils/fp8_utils.py

LucasWilkinson

LGTM now, thanks!

Signed-off-by: Shu Wang <shuw@nvidia.com>

) Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

) Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

) Signed-off-by: Shu Wang <shuw@nvidia.com>

) Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

mergify bot added the ci/build label Mar 6, 2025

wenscarl force-pushed the scale_mm_bw_sm100 branch from 2f5591d to 546b495 Compare March 10, 2025 20:00

wenscarl changed the title ~~[wip] scaled_mm_bw_sm100~~ Scaled_mm_blockwise_sm100 support by cutlass Mar 10, 2025

wenscarl marked this pull request as ready for review March 10, 2025 20:03

wenscarl requested a review from tlrmchlsmth as a code owner March 10, 2025 20:03

wenscarl changed the title ~~Scaled_mm_blockwise_sm100 support by cutlass~~ Add cutlass support for blackwell fp8 blockwise gemm Mar 10, 2025

wenscarl force-pushed the scale_mm_bw_sm100 branch from 3a34259 to 07e8083 Compare March 21, 2025 03:25

WoosukKwon assigned tlrmchlsmth Mar 26, 2025

tlrmchlsmth reviewed Mar 26, 2025

View reviewed changes

mergify bot added the needs-rebase label Mar 28, 2025

wenscarl requested review from WoosukKwon, mgoin and robertgshaw2-redhat as code owners March 28, 2025 18:44

wenscarl force-pushed the scale_mm_bw_sm100 branch from 287cfb3 to ca3a3e2 Compare March 28, 2025 18:47

wenscarl requested a review from tlrmchlsmth March 28, 2025 18:48

wenscarl force-pushed the scale_mm_bw_sm100 branch from ca3a3e2 to 87d109d Compare March 28, 2025 21:55

mergify bot removed the needs-rebase label Mar 28, 2025

wenscarl force-pushed the scale_mm_bw_sm100 branch 2 times, most recently from a06d3da to 3dfa546 Compare March 30, 2025 16:20

tlrmchlsmth reviewed Apr 4, 2025

View reviewed changes

csrc/cutlass_extensions/common.hpp Outdated Show resolved Hide resolved

tlrmchlsmth approved these changes Apr 4, 2025

View reviewed changes

thakkarV reviewed Apr 4, 2025

View reviewed changes

csrc/quantization/cutlass_w8a8/c3x/scaled_mm_blockwise_sm100_fp8_dispatch.cuh Outdated Show resolved Hide resolved

csrc/quantization/cutlass_w8a8/c3x/scaled_mm_blockwise_sm100_fp8_dispatch.cuh Outdated Show resolved Hide resolved

mergify bot added the needs-rebase label Apr 23, 2025

wenscarl force-pushed the scale_mm_bw_sm100 branch 2 times, most recently from 703ec2c to 1d17dd1 Compare April 25, 2025 17:19

mergify bot removed the needs-rebase label Apr 25, 2025

wenscarl force-pushed the scale_mm_bw_sm100 branch from 1d17dd1 to 92d6da8 Compare April 25, 2025 17:23

wenscarl requested a review from thakkarV April 25, 2025 18:54

wenscarl force-pushed the scale_mm_bw_sm100 branch from 92d6da8 to 86d58fd Compare April 25, 2025 19:05

LucasWilkinson reviewed Apr 25, 2025

View reviewed changes

vllm/model_executor/layers/quantization/utils/fp8_utils.py Outdated Show resolved Hide resolved

LucasWilkinson reviewed Apr 25, 2025

View reviewed changes

vllm/model_executor/layers/quantization/utils/fp8_utils.py Outdated Show resolved Hide resolved

wenscarl force-pushed the scale_mm_bw_sm100 branch from 7ca702c to f207fec Compare May 5, 2025 02:29

wenscarl requested a review from LucasWilkinson May 5, 2025 02:43

LucasWilkinson approved these changes May 5, 2025

View reviewed changes

LucasWilkinson enabled auto-merge (squash) May 5, 2025 13:56

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label May 5, 2025

auto-merge was automatically disabled May 6, 2025 18:42
Head branch was pushed to by a user without write access

wenscarl force-pushed the scale_mm_bw_sm100 branch from f207fec to cc21ba5 Compare May 6, 2025 18:42

wenscarl added 7 commits May 6, 2025 19:16

Support cutlass w8a8 blockwise matmul for blackwell

d001b23

Signed-off-by: Shu Wang <shuw@nvidia.com>

Update CUDA_VERSION requirement.

b45e6bb

Signed-off-by: Shu Wang <shuw@nvidia.com>

Improve after review 1

5100a33

Signed-off-by: Shu Wang <shuw@nvidia.com>

Correct forward compatibility.

24a528e

Signed-off-by: Shu Wang <shuw@nvidia.com>

Branching with 1/2SMs

c030b94

Signed-off-by: Shu Wang <shuw@nvidia.com>

Add comments for multiple of 4 constraint

9bab9f6

Signed-off-by: Shu Wang <shuw@nvidia.com>

Fix precommit.

b6783db

Signed-off-by: Shu Wang <shuw@nvidia.com>

wenscarl force-pushed the scale_mm_bw_sm100 branch from cc21ba5 to b6783db Compare May 6, 2025 19:18

simon-mo merged commit 376786f into vllm-project:main May 8, 2025
77 of 80 checks passed

princepride pushed a commit to princepride/vllm that referenced this pull request May 10, 2025

Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383

765df60

) Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383

5725ea8

) Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025

Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383

ce8f7fb

) Signed-off-by: Shu Wang <shuw@nvidia.com>

zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025

Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383

e459097

) Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

Uh oh!

Conversation

wenscarl commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

pathorn commented Mar 6, 2025

Uh oh!

kushanam commented Mar 10, 2025

Uh oh!

WoosukKwon commented Mar 25, 2025

Uh oh!

tlrmchlsmth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Mar 28, 2025

Uh oh!

Uh oh!

tlrmchlsmth commented Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

LucasWilkinson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wenscarl commented Mar 6, 2025 •

edited

Loading