Repository navigation
[Perf][Kernel] Integrate Mega-mHC from DeepGEMM for DeepSeek V4.1 (reopen of #56255) - #56962
Merged
ywang96 merged 8 commits intoSep 15, 2026
Merged
Conversation
…ll convention Repoint the DeepGEMM pin from deepseek-ai/DeepGEMM nv_dev (2.6.1) to vllm-project/DeepGEMM dev at e49b29bb (2.8.0). The fork carries the SM120 port we relied on nv_dev for, plus configurable SwiGLU alpha/beta and the SiTU activation for FP8/FP4 Mega MoE. 2.8.0 renames the Mega MoE activation parameters: the optional situ_beta/situ_linear_beta pair becomes activation_alpha/activation_beta (plain floats, defaulting to 1.0/0.0). For SiTU, activation_alpha is the gate tanh scale and activation_beta the linear/up tanh scale, where 0.0 leaves the up branch untouched. Update the Kimi K3 MegaMoE call site accordingly. The SwiGLU defaults reproduce the previous behavior, so the DeepSeek V4 call sites are unchanged. 2.8.0 also swaps the third-party/fmt submodule for third-party/deep_jit, so update the submodule list, the _C include path, and the header glob that retriggers the _C rebuild. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
DeepGEMM 2.8.0 removes the legacy `fp8_m_grouped_gemm_nt_masked` Python alias, so the getattr lookup silently resolved to None and every call raised "DeepGEMM backend is unavailable" — breaking BatchedDeepGemmExperts. Look up the canonical `m_grouped_fp8_gemm_nt_masked` first and keep the legacy spelling as a fallback for externally pinned older DeepGEMM. Both names alias the same `m_grouped_fp8_fp4_gemm_nt_masked` binding, whose signature and pybind defaults are unchanged across the two versions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
The fork commit this was first pinned to, 9a86ae2b, re-ported the SM90 paged-MQA kv_block=32/next_n=4 support onto the 2.8.0 base but dropped the zero-context-length guards that deepseek-ai's nv_dev branch carried. With all context lengths zero the metadata kernel reads prefix_sum[batch_size], one past the end of a shared buffer sized to exactly align(batch_size, 32) ints, and traps with CUDBG_EXCEPTION_WARP_OUT_OF_RANGE_ADDRESS. The DSA indexer warmup runs a dummy batch of max_num_seqs (default 1024, already 32-aligned) rows with all context lengths zero, so every DeepSeek V4 startup on Hopper hit it: nvidia-h100-lm-eval-kv-offload-large failed deterministically on all 4 ranks of both test_gsm8k_offloading_correctness cases. Move the pin to the dev tip, which now carries vllm-project/DeepGEMM#8 restoring the four guards and adding a zero-context regression test. Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg> (cherry picked from commit 74c2daf) Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Rebase the closed vllm-project#56255 Mega-mHC integration on the vllm-project#56876 dependency base, retargeting deepseek_v41. Preserve the TileLang fallback and Engram auxiliary-state semantics, and extend correctness and CUDA graph coverage. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Juntian777
requested review from
AndreasKaratzas,
Harry-Chen,
LucasWilkinson,
WoosukKwon,
khluu,
mgoin,
tlrmchlsmth,
yewentao256 and
zyongye
as code owners
September 15, 2026 06:44
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
|
✅ @Juntian777, CI is now available for this PR.
|
Member
|
/ci run |
|
✅ Triggered Buildkite CI #89077 for commit |
ywang96
force-pushed
the
integrate/pr56255-mega-mhc
branch
from
September 15, 2026 10:58
6b53657 to
91368b7
Compare
gcanlin
added a commit
to gcanlin/vllm
that referenced
this pull request
Sep 15, 2026
Upstream replaced the fused TileLang post+pre call sites with the DeepGEMM Mega-mHC wrapper mhc_shifted_post_pre (vllm-project#56962); this branch had wrapped the same call sites in the mhc_stream split path. Resolved by keeping both: the split path when a coefficient stream is active, mhc_shifted_post_pre otherwise. Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
keneoneth
pushed a commit
to keneoneth/vllm
that referenced
this pull request
Sep 16, 2026
…open of vllm-project#56255) (vllm-project#56962) Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Signed-off-by: Juntian Liu <Juntianl777@gmail.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
17 of 19 tasks
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
1b5a7abb58) with the model path retargeted todeepseek_v41.ad1f1726(2.8.0), including its SM90/SM120 support and API fixes.Why this is not a duplicate
#56255 was closed without merging; this PR reopens that work. Open-PR searches show no competing Mega-mHC integration. #56611 fuses TileLang statistics with attention/FFN instead; this change uses DeepGEMM Mega-mHC fusion. Dependency changes intentionally reuse #56876 rather than re-opening them.
Benchmarks
Single GB200, H=5120 / 7168,
hc_mult=4, BF16, 20 Sinkhorn iterations. Full post + delayed-pre + RMSNorm chain. CUPTI full-stage spans over CUDA graph replays, cold L2, 10 warmup + 100 samples per variant in forward/reverse (ABCCBA) order. Setup, JIT, and host allocation excluded; CPU launch overhead is outside the measured spans.E2E Benchmarks
GB200 TP4, official DeepSeek-V4.1-Flash,
fp8_ds_mla, no speculative decoding,max-model-len=40960. Same local source (e7fcd569dd) and dependencies; TileLang uses the existing fused-op worker hook, and #56611 Overlap is ported onto this base with TP4 enabled.vllm bench serve: exactly 8192/32768 input + 1024 output tokens, seed 0, temperature 0, one warmup, ignore EOS, prefix caching disabled. C1: 20 requests/case; C32: 64. All 504 requests completed with exact token counts.Latencies are in ms. Δ is relative to TileLang, shown as mean TPOT / output throughput; negative TPOT and positive throughput are better.
Concurrency 1:
Concurrency 32:
Conclusion: Mega-mHC and Overlap have essentially equal C1 TPOT; Mega has lower TTFT. At C32, Mega delivers 3.7% / 5.0% more output tok/s than Overlap at 8k / 32k.
Tests
.venv/bin/python -m pytest tests/kernels/test_mhc_kernels.py -q -x: 174 passed, 32 skipped — covers empty/decode/prefill and changed-input CUDA graph replay. FP8 DeepGEMM tests: 100 passed, 25 skipped; matching-extension qnorm/RoPE tests: 6 passed. Pre-commit checks passed.Both accuracy evaluations used Mega-mHC with the official
deepseek-ai/DeepSeek-V4.1-Flashcheckpoint, on exclusive GB200 nodes with TP4+EP,fp8_ds_mla, DeepGEMM MegaMoE, CPU-offloaded Engram, no speculative decoding, and prefix caching disabled. One pass run at concurrency 32, request seed 0 and dataset/few-shot seed 1234. GSM8K: 1319 problems, 5-shot, completions endpoint, greedy (temperature=0,top_p=1), max output 4096 and server context 8192; strict-match and flexible-extract both scored 1225/1319 (92.87%), with one output-limit hit. GPQA Diamond: 198 problems, 0-shot, chat completions,think=true, defaulthighreasoning effort (not explicitly overridden),temperature=1,top_p=0.95, max output 262144 and server context 1048576; 179/198 (90.40%), with no output-limit hits. Both had zero request errors; server max batched tokens was 8192, max sequences 256.AI assistance
This change was developed with AI assistance (OpenAI Codex); all changes were reviewed and tested by the human submitter.