Skip to content

Add a flag to use FusedMoE kernel in compressed quantization #23442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chenxi-yang wants to merge 2 commits into from
Closed

Add a flag to use FusedMoE kernel in compressed quantization #23442

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
996dc60
Add a flag to use FusedMoE kernel in compressed quantizationon on sm9…
Sep 24, 2025
74a1502
Merge branch 'main' into export-D80552023
chenxi-yang Sep 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -542,11 +542,12 @@ def __init__(
self.rocm_aiter_moe_enabled = is_rocm_aiter_moe_enabled()

# cutlass path
self.is_fp8_w8a8_sm90 = quant_config._is_fp8_w8a8_sm90(
self.weight_quant, self.input_quant)
self.is_fp8_w8a8_sm100 = quant_config._is_fp8_w8a8_sm100(
self.weight_quant, self.input_quant)
self.use_cutlass = not self.block_quant and (
quant_config._is_fp8_w8a8_sm90(self.weight_quant, self.input_quant)
or self.is_fp8_w8a8_sm100)
self.use_cutlass = not self.block_quant and (self.is_fp8_w8a8_sm90
or self.is_fp8_w8a8_sm100)
self.disable_expert_map = False

def create_weights(self, layer: torch.nn.Module, num_experts: int,
Expand Down Expand Up @@ -1013,8 +1014,9 @@ def apply(
elif self.use_cutlass:
assert self.moe_quant_config is not None

# small-batch fallback on SM100
if self.is_fp8_w8a8_sm100 and topk_ids.shape[0] <= 8:
# SM90 or small-batch fallback on SM100
if self.is_fp8_w8a8_sm90 or (self.is_fp8_w8a8_sm100
and topk_ids.shape[0] <= 8):
from vllm.model_executor.layers.fused_moe import fused_experts
assert per_act_token == per_channel_quant
return fused_experts(
Expand Down
Loading