Repository navigation
Conversation
… ROCm Squash of the MiniMax-M3.1 serving branch, rebased on 5d35ce3. NVIDIA: - Route NVFP4 routed experts in MXFP8 checkpoints to the CT NVFP4 MoE - NVFP4 KV cache on the MSA sparse attention path, served only by the CUTLASS Q8KV4 kernels (E4M3 query) on SM100/SM103/SM107 - NVFP4 index-K cache via the MSA Q8KV4/Q8KV8 indexers; opt-in decode context parallelism for the NVFP4 indexer at TP4 - Vendor fmha_sm100 from vllm-project/MSA dev 25c1447 - NVFP4 MSA decode past 512 sequences: Q8KV4 limits, bounded indexer and plan memory - Decode metadata before the forward: 13 ops -> 2; in-place indexer I/O, no counter reset, PDL chain for the NVFP4 sparse decode - DeepGEMM NVFP4 MegaMoE on the unified DeepGemmMegaMoEExperts layer; per-token FC1 input scales outside MegaMoE; split staging rows over programs - Fuse small-batch MoE finalize into the all-reduce + Gemma norm + MXFP8 quant - QKV+indexer MXFP8 GEMM with fused epilogue; SM100 fused dense GEMM1 - Fused QK-norm/RoPE/KV insert: persistent row-group kernel - Read aux hidden states after the deferred TP all-reduce - FA4: pass no FP8 descale for a unit per-tensor scale - Keep KV cache zeroing for mixed-precision UniformTypeKVCacheSpecs groups - Clear per-token A4 metadata for NVFP4 W4A16 MoE kernels (Marlin only) - Hand the Q8KV4 prefill a per-device split workspace - Align image preprocessing: long-side resize with detail tiers, fp32 patch embedding - NIXL KV-head re-sharding for per-head-contiguous NVFP4 pages; BLHNC layout for NIXL transfer - SimpleCPUOffload: 1 GiB HugeTLB host buffer + PreferCE batch copies Spec decode: - DSpark for MiniMax-M3.1: draft model, fixes and perf; default the draft to the checkpoint's dspark/ subfolder - Fuse the draft's q/k norm + RoPE and its context-K/V write - FLASHINFER_TS: prims_ts paged decode for non-causal draft blocks - Store the draft tokens' K/V under torch.compile - Markov sampling: one step helper, bias in the Gumbel kernel - DFlash: pad the prepared inputs in wide blocks on every platform ROCm: - Re-quantize the NVFP4 routed experts to MXFP8 (or MXFP4) at load; NVFP4 -> MXFP4 re-quantization and fused shared experts through the online quantization API - Mono decode (one FlyDSL launch per layer): native MXFP8 shared expert, top-16 sort/merge, aux hidden states, indexer CP - NVFP4 KV + NVFP4 indexer on gfx950, decode indexer CP - Tune small-M MXFP8 linears - Sequence-parallel prefill under pure TP (opt-in), with MXFP4 MoE support; TBO/DBO under pure TP; warm the microbatch threads' hipBLASLt handles at init - gfx942: avoid the load-time FP8 MoE quant crash; fall back from the fused QK-norm - Keep the NVFP4 KV-insert helpers CUDA-only so the fused kernel builds on HIP Co-authored-by: Taoyu Zhu <zhutaoyu97@gmail.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Co-authored-by: qizixi <22851944+zixi-qi@users.noreply.github.com> Co-authored-by: indivats <Indira.Vats@amd.com> Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com> Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: fxmarty-amd <felmarty@amd.com> Co-authored-by: Jiejing Zhang <280536569+jiejingzhangamd@users.noreply.github.com> Co-authored-by: Prasun Gera <pgera@users.noreply.github.com> Co-authored-by: Dao Le <daole@inferact.ai> Co-authored-by: Isotr0py <mozf@inferact.ai> Co-authored-by: Neha Prakriya <neha.prakriya@amd.com> Co-authored-by: Summer Yang <girasoley@inferact.ai> Co-authored-by: Zhewen Li <jerven.vllm@gmail.com> Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com> Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com> Co-authored-by: v <v@inferact.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
…wn PRs (vllm-project#63490) Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ssage (vllm-project#63492) Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
Without shared-expert fusion, have the router fold routed_scaling_factor into the top-k weights instead of scaling the bf16 MoE output, and pass the factor to AITER's biased_grouped_topk, which applies it in-kernel, rather than multiplying the weights afterwards. Removes one elementwise kernel per MoE layer; the weights are bit-identical. Signed-off-by: Micah Williamson <micah.williamson@amd.com>
micah-wil
force-pushed
the
m31-moe-scale-fold
branch
from
October 28, 2026 05:11
0b9cb7b to
41a476c
Compare
zyongye
force-pushed
the
minimax-m31-flash
branch
from
October 29, 2026 07:00
a0ea941 to
d0ebc06
Compare
mergify
Bot
added
ci/build
deepseek
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
zyongye
force-pushed
the
minimax-m31-flash
branch
3 times, most recently
from
October 29, 2026 17:08
1f5ea52 to
f4ca759
Compare
khluu
force-pushed
the
minimax-m31-flash
branch
from
October 29, 2026 20:59
f4ca759 to
f4bf304
Compare
2 of 6 tasks
zyongye
force-pushed
the
minimax-m31-flash
branch
from
October 31, 2026 01:27
f4bf304 to
3e48944
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Without shared-expert fusion, have the router fold routed_scaling_factor into the top-k weights instead of scaling the bf16 MoE output, and pass the factor to AITER's biased_grouped_topk, which applies it in-kernel, rather than multiplying the weights afterwards. Removes one elementwise kernel per MoE layer
Overview
Claims
Validation
Details
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)