Port DeepSeek Sparse Attention to `MambaModel` by janEbert · Pull Request #3553 · NVIDIA/Megatron-LM

janEbert · 2026-02-24T00:15:05Z

What does this PR do ?

Make experimental DeepSeek Sparse Attention (DSA) available to MambaModel. For now, we error out when a user tries to use both standard Attention and DSA. It's not supported right now to have different Attention flavors in the same hybrid model, this requires broader restructuring.

Pre-checks

I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

(Step 1): Add PR label `Expert Review`

(Step 2): Collect the expert reviewers reviews

Attach the Expert Review label when your PR is ready for review.
GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

Add Final Review label
GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

copy-pr-bot · 2026-02-24T00:16:41Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

duncanriach

Quick review.

I want this to merge after 3377. This will need to be adjusted to accommodate the changes in that PR

duncanriach · 2026-02-24T21:13:31Z

Ugh, yeah. Checkpoint compatibility is an issue

janEbert · 2026-02-25T08:59:45Z

Thank you for the quick review, Duncan! Should I maybe already start rebasing this on top of #3377?

duncanriach · 2026-02-25T18:12:00Z

Thank you for the quick review, Duncan! Should I maybe already start rebasing this on top of #3377?

Maybe good to hold off a bit since there might be some more changes in the PR now. Quite a lot of feedback just came in, including from you :-)

janEbert · 2026-03-03T14:45:28Z

/ok to test bcb6b4f

Phlip79 · 2026-04-16T19:15:32Z

/ok to test 769704d

svcnvidia-nemo-ci · 2026-04-16T20:49:12Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24533263668

Phlip79 · 2026-04-16T22:13:02Z

/ok to test 931ca7c

ko3n1g · 2026-04-17T06:56:31Z

/ok to test bb32183

Also prefer modern type hint specification using native types.

And add corresponding tests. DSA = DeepSeek Sparse Attention

By using a generalized dictionary with the same sorting as in the hardcoded return dict, we get future-compatibilty in case other layers are added.

By using a dictionary with the same sorting as in the hardcoded list used to create the returned list (the specified return type of tuple is incorrect), we get cleaner code.

Concerns the `get_layer_maps_from_layer_type_list` method. This seems cleaner.

DSA = DeepSeek Sparse Attention

- New pytest test `test_dsa_gpt_mamba_equivalence.py` builds both a GPTModel (DSA, 4 layers) and a MambaModel (pattern S-S-S-S-, 8 layers) in-memory, remaps weights GPT→Mamba, and asserts logprob equivalence across TP=1/PP=1, TP=2/PP=1, and TP=1/PP=2 distributed configs. - New checkpoint conversion utility `tools/checkpoint/remap_gpt_dsa_to_mamba.py` applies the same layer-key remapping (decoder.layers.{N} → {2N}/{2N+1}, decoder.final_layernorm → decoder.final_norm) to DCP checkpoints.

Extend the DSA GPT/Mamba logprob equivalence suite to cover mixed dense+MoE architectures, mirroring the real DeepSeek-V3 layout where the first N layers are dense and the remaining layers use MoE. Key changes: - Add `pre_mlp_layernorm.*` routing in `_remap_gpt_to_mamba_state_dict` and `_remap_key` (checkpoint tool): MoE layers expose a real TENorm for `pre_mlp_layernorm` (not fused), which maps to MoETransformerLayer 2N+1. Dense layers use IdentityOp and produce no keys, so existing tests are unaffected. - Add `_make_dsa_moe_config` with `moe_layer_freq=[0,0,1,1]` (first 2 GPT layers dense, last 2 MoE) and proxy MoE params matching the DeepSeek-V3 style (4 experts, grouped-gemm, allgather dispatcher, shared experts). - Add `_MOE_MAMBA_PATTERN = "S-S-SESE"` and `TestDSAMoEGPTMambaEquivalence` with the same three parametrized tests as the dense suite (tp=1/2 pp=1/2): logprob match, strict weight loading, and golden-value recording/comparison.

janEbert · 2026-04-17T12:29:38Z

/ok to test 6fc1c7c

svcnvidia-nemo-ci · 2026-04-17T13:33:58Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24567828712

svcnvidia-nemo-ci · 2026-04-17T13:40:59Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24568144051

janEbert requested review from a team as code owners February 24, 2026 00:15

svcnvidia-nemo-ci requested a review from a team February 24, 2026 00:15

svcnvidia-nemo-ci added this to the Core 0.16 milestone Feb 24, 2026

copy-pr-bot Bot temporarily deployed to test February 24, 2026 00:15 Inactive

janEbert marked this pull request as draft February 24, 2026 00:16

janEbert force-pushed the mamba-dsa branch from 0249ad8 to c85dc05 Compare February 24, 2026 18:35

duncanriach reviewed Feb 24, 2026

View reviewed changes

janEbert force-pushed the mamba-dsa branch from c85dc05 to 1e116b0 Compare February 25, 2026 12:55

janEbert force-pushed the mamba-dsa branch 11 times, most recently from de61e61 to bcb6b4f Compare March 3, 2026 14:40

copy-pr-bot Bot temporarily deployed to test March 3, 2026 14:46 Inactive

janEbert force-pushed the mamba-dsa branch 3 times, most recently from df1bfb7 to 27f7c8a Compare March 4, 2026 17:46

janEbert force-pushed the mamba-dsa branch from b09a33f to 31b69e4 Compare April 9, 2026 21:02

copy-pr-bot Bot temporarily deployed to test April 9, 2026 21:03 Inactive

Phlip79 removed the Expert Review [deprecated] Apply this label to indicate that your PR is ready for expert review. label Apr 9, 2026

Phlip79 removed the request for review from a team April 15, 2026 19:05

jaredcasper approved these changes Apr 16, 2026

View reviewed changes

Comment thread megatron/core/models/mamba/mamba_model.py

svcnvidia-nemo-ci added Approved All necessary approvals have been made and removed Final Review PR is in the "final review" stage labels Apr 16, 2026

janEbert force-pushed the mamba-dsa branch from 31b69e4 to 769704d Compare April 16, 2026 19:09

janEbert requested a review from a team as a code owner April 16, 2026 19:09

janEbert enabled auto-merge April 16, 2026 19:10

janEbert added 12 commits April 17, 2026 14:28

Fix Mamba MLA handling

b004e05

Fix return type

786c50f

Also prefer modern type hint specification using native types.

Improve error type

bfab53e

Fix DSA dispatch

67b145c

And add corresponding tests. DSA = DeepSeek Sparse Attention

Port DeepSeek Sparse Attention to MambaModel

5b5a328

Future-proof get_hybrid_layer_counts return dict

69d7742

By using a generalized dictionary with the same sorting as in the hardcoded return dict, we get future-compatibilty in case other layers are added.

Prefer dict to list/tuple for large return values

44d344e

By using a dictionary with the same sorting as in the hardcoded list used to create the returned list (the specified return type of tuple is incorrect), we get cleaner code.

Do not treat DSA layers as Attention layers

25dc43b

Concerns the `get_layer_maps_from_layer_type_list` method. This seems cleaner.

Handle hybrid standard Attention/MLA models

75bbc7f

Add DSA Mamba tests

4258b92

DSA = DeepSeek Sparse Attention

janEbert mentioned this pull request Apr 23, 2026

Port Multi-Latent Attention to HybridModel #4452

Merged

Uh oh!

Conversation

janEbert commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Pre-checks

Code review

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

(Step 3): Final Review

(Optional Step 4): Cherry-pick into release branch

Merging your PR

Uh oh!

copy-pr-bot Bot commented Feb 24, 2026

Uh oh!

duncanriach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

duncanriach Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

janEbert commented Feb 25, 2026

Uh oh!

duncanriach commented Feb 25, 2026

Uh oh!

janEbert commented Mar 3, 2026

Uh oh!

Uh oh!

Phlip79 commented Apr 16, 2026

Uh oh!

svcnvidia-nemo-ci commented Apr 16, 2026

Uh oh!

Phlip79 commented Apr 16, 2026

Uh oh!

ko3n1g commented Apr 17, 2026

Uh oh!

janEbert commented Apr 17, 2026

Uh oh!

svcnvidia-nemo-ci commented Apr 17, 2026

Uh oh!

svcnvidia-nemo-ci commented Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

janEbert commented Feb 24, 2026 •

edited

Loading

(Step 1): Add PR label `Expert Review`