[Perf] Deepseek V41 Attention MegaKernel - #56344

Closed

zyongye wants to merge 28 commits into

dsv41-optimizedfrom

v41-megakernel

Commits on Sep 10, 2026

[Docs] DeepSeek V4.1 megakernel integration spec and implementation plan
zyongye
committed
29822bc
[DSv4.1] Add fused-kernel layout permutation helpers
zyongye
committed
728f4cd
[Attention] Expose int32 positions in the DeepSeek sparse SWA metadata
zyongye
committed
f9bf66e
[Config] Add DeepSeek V4.1 fused attention flags
zyongye
committed
acd13c9
[Build] Pin FlashMLA to upstream deepseek-ai/FlashMLA at the DeepSeek V4.1 kernels commit
zyongye
committed
c820250
[Attention] Wrap the FlashMLA fused sparse attention ops
zyongye
committed
cc99b66
[DSv4.1] Add the fused-layout Q padding kernel
zyongye
committed
04b3514
[DSv4] Let fused_inv_rope_fp8_quant emit the fused-kernel chunk layout
zyongye
committed
a3e76dd
[DSv4.1] Factor the attention output buffer and o_proj tail into hooks
zyongye
committed
263632e
[DSv4.1] Add the FlashMLA fused sparse attention layer
zyongye
committed
ba3b026
[Bench] Compare the FlashMLA fused sparse kernel with the split-KV pipeline
zyongye
committed
d07b715
[DSv4.1] Add the nvfp4_ds_mla KV cache layout (V4.1 fp8 SWA + V4.1 fp4 compressed)
zyongye
committed
dc441b5
[Config] Default DeepSeek V4.1 VL wrapper to breakable CUDA graphs
zyongye
committed
b8863fb
[Docs] Record implementation status and Phase 0 results for the V4.1 megakernel work
zyongye
committed
8253724
[DSv4.1] Match fused-attention weight permutation on child-relative names
zyongye
committed
b30aabe
[Docs] Note the wrapper-prefix permutation bug and LLM-path environment fixes
zyongye
committed
40cb457
[Docs] Record fused vs unfused end-to-end results
zyongye
committed
5917280
docs: record nvfp4_ds_mla end-to-end smoke result in the megakernel spec
zyongye
committed
bb54c56
docs(plan): check off the fp8 and nvfp4 local end-to-end smoke steps

zyongye
and
claude
committed
d2f58c9
fix(dsv41): write -1 sentinels past combined_lens in combine_topk_swa_indices
zyongye
committed
451b930
docs: record the gsm8k regression root cause and bs1 fused latency in the spec

zyongye
and
claude
committed
aa7bb31
docs: record post-fix gsm8k parity for the fused DSv4.1 attention

zyongye
and
claude
committed
80b7d1f
test: stub the renamed _prepare_q_and_insert_kv hook in the compressor cache test

zyongye
and
claude
committed
f6c8f6c
docs: record the post-fix nvfp4_ds_mla mini gsm8k result

zyongye
and
claude
committed
5a53595
docs: record the same-branch bs1 A/B and kernel benchmark for the fused layer

zyongye
and
claude
committed
c87d77b
docs: record the full nvfp4_ds_mla gsm8k parity run and remaining open items

zyongye
and
claude
committed
cd3b0a4
Drop development tests, design docs and test-only layout helpers

zyongye
and
claude
committed
9ed2746
[Build] Pin FlashMLA to vllm-project/FlashMLA c112cc1 (V4.1 kernels synced)

zyongye
and
claude
committed
8c2d92c

Navigation Menu

Uh oh!

[Perf] Deepseek V41 Attention MegaKernel - #56344

[Perf] Deepseek V41 Attention MegaKernel#56344
zyongye wants to merge 28 commits into
dsv41-optimizedfrom
v41-megakernel

Commits on Sep 10, 2026

[Docs] DeepSeek V4.1 megakernel integration spec and implementation plan

[DSv4.1] Add fused-kernel layout permutation helpers

[Attention] Expose int32 positions in the DeepSeek sparse SWA metadata

[Config] Add DeepSeek V4.1 fused attention flags

[Build] Pin FlashMLA to upstream deepseek-ai/FlashMLA at the DeepSeek V4.1 kernels commit

[Attention] Wrap the FlashMLA fused sparse attention ops

[DSv4.1] Add the fused-layout Q padding kernel

[DSv4] Let fused_inv_rope_fp8_quant emit the fused-kernel chunk layout

[DSv4.1] Factor the attention output buffer and o_proj tail into hooks

[DSv4.1] Add the FlashMLA fused sparse attention layer

[Bench] Compare the FlashMLA fused sparse kernel with the split-KV pipeline

[DSv4.1] Add the nvfp4_ds_mla KV cache layout (V4.1 fp8 SWA + V4.1 fp4 compressed)

[Config] Default DeepSeek V4.1 VL wrapper to breakable CUDA graphs

[Docs] Record implementation status and Phase 0 results for the V4.1 megakernel work

[DSv4.1] Match fused-attention weight permutation on child-relative names

[Docs] Note the wrapper-prefix permutation bug and LLM-path environment fixes

[Docs] Record fused vs unfused end-to-end results

docs: record nvfp4_ds_mla end-to-end smoke result in the megakernel spec

docs(plan): check off the fp8 and nvfp4 local end-to-end smoke steps

fix(dsv41): write -1 sentinels past combined_lens in combine_topk_swa_indices

docs: record the gsm8k regression root cause and bs1 fused latency in the spec

docs: record post-fix gsm8k parity for the fused DSv4.1 attention

test: stub the renamed _prepare_q_and_insert_kv hook in the compressor cache test

docs: record the post-fix nvfp4_ds_mla mini gsm8k result

docs: record the same-branch bs1 A/B and kernel benchmark for the fused layer

docs: record the full nvfp4_ds_mla gsm8k parity run and remaining open items

Drop development tests, design docs and test-only layout helpers

[Build] Pin FlashMLA to vllm-project/FlashMLA c112cc1 (V4.1 kernels synced)