Skip to content

[MoE] Accept prequantized DeepGEMM inputs - #51941

Draft
WoosukKwon wants to merge 1 commit into
vllm-project:mainfrom
WoosukKwon:agent/deepgemm-moe-prequant
Draft

WoosukKwon wants to merge 1 commit into
vllm-project:mainfrom
WoosukKwon:agent/deepgemm-moe-prequant

Conversation

@WoosukKwon

Copy link
Copy Markdown
Collaborator

Purpose

Allow modular FP8 MoE execution to consume a matching packed DeepGEMM
activation produced upstream, avoiding duplicate input quantization.

This PR adds an explicit prequantized-input capability to modular prepare/finalize
and expert interfaces, transports the data and scale through the opaque MoE op,
and teaches DeepGEMM's expert scatter to accept already-packed UE8M0 scales.
Unsupported dispatch modes retain the existing path.

This is independent of the block-linear consumer in #51939 and does not include
model-level all-reduce or MTP changes.

Duplicate-work check

No issue number was provided. I searched open PRs for MoE prequantized input
and DeepGEMM prequantized activation; the only exact prior implementation was
the superseded draft #51936. #49636 concerns a FlashInfer MoE-EP backend and
does not add this modular DeepGEMM input contract.

Tests

.venv/bin/python -m pytest \
  tests/kernels/moe/test_deepgemm.py \
  tests/kernels/moe/test_silu_mul_fp8_quant_deep_gemm.py -q
# 32 passed, 3 skipped

git diff --cached --name-only -z | xargs -0 pre-commit run --files
# all applicable hooks passed, including ruff and mypy

AI assistance disclosure

This change was developed with OpenAI Codex assistance. This is a draft PR;
the human submitter must review every changed line and confirm they understand
and can defend the change before marking it ready.

Propagate matching packed FP8 activations through modular MoE dispatch and let DeepGEMM skip duplicate input quantization.

Co-authored-by: OpenAI Codex <noreply@openai.com>

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
@mergify

mergify Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @WoosukKwon.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant