Skip to content

[Perf] dsv4: Use the fused MXFP8 producers in eager prefill #45903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hnyls2002 merged 1 commit into from
Oct 22, 2026
Merged

[Perf] dsv4: Use the fused MXFP8 producers in eager prefill #45903

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions python/sglang/srt/models/deepseek_v4_mhc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@
from sglang.srt.layers.quantization.mxfp8_input import Mxfp8SwizzledInput
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
from sglang.srt.model_executor.runner import compile_in_capture_mode
from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph.context import (
is_in_breakable_cuda_graph,
)
from sglang.srt.models.deepseek_v2 import MoEOutput
from sglang.srt.runtime_context import get_forward, get_parallel, get_platform
from sglang.srt.utils import is_gfx95_supported, is_hip, is_npu, is_xpu
Expand All @@ -54,11 +57,14 @@


def fused_mxfp8_rows(rows: int) -> bool:
"""Decode and verify batches the fused MXFP8 producers cover."""
"""Batches the fused MXFP8 producers cover: decode and verify, and eager
prefill (not inside a breakable prefill graph)."""
fwd = get_forward()
return 0 < rows <= FUSED_MXFP8_PRODUCER_MAX_ROWS and not (
fwd.is_extend_in_batch or fwd.sp_active
)
if rows <= 0 or fwd.sp_active:
return False
if fwd.is_extend_in_batch:
return not is_in_breakable_cuda_graph()
return rows <= FUSED_MXFP8_PRODUCER_MAX_ROWS


def empty_deepgemm_mxfp8_pair(
Expand Down
Loading