Skip to content

Fix: interleaved RoPE application for MLA and Support Index Cache DSA indexer skip-topk sharing for GLM5 - #46372

Merged
ArthurZucker merged 12 commits into
huggingface:mainfrom
Dovis01:fix-glm-moe-dsa
Jun 8, 2026
Merged

Fix: interleaved RoPE application for MLA and Support Index Cache DSA indexer skip-topk sharing for GLM5#46372
ArthurZucker merged 12 commits into
huggingface:mainfrom
Dovis01:fix-glm-moe-dsa

Conversation

@Dovis01

@Dovis01 Dovis01 commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Fix interleaved RoPE for MLA and add MoE/DSA indexer cache to accelerate Sparse Attention

Summary

This PR fixes several correctness issues in GlmMoeDsa around RoPE application. It also adds DSA indexer cache / skip-topk support for GLM5: on shared layers, attention reuses the previous full indexer layer’s top-k indices instead of recomputing them, reducing repeated top-k computation and accelerating Sparse Attention.

This feature is inspired by the index cache technique described in this paper Index cache, and adapts the idea to the GLM5 MoE/DSA attention path.

Dovis01 added 7 commits June 2, 2026 08:00
@Dovis01
Fix error attributes mapping
67a9994 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Fix tests
3f02fef 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Merge branch 'main' into fix-glm-moe-dsa
3841bbc
@Dovis01
Merge branch 'main' into fix-glm-moe-dsa
386e43f
@Dovis01
Merge branch 'main' into fix-glm-moe-dsa
e59ee7c 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Fix: Support interleaved RoPE for MLA
23d7895 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Fix tests
8eab52b 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Dovis01 force-pushed the fix-glm-moe-dsa branch from dd86315 to 8eab52b Compare June 3, 2026 09:54
@Rocketknight1

Rocketknight1 commented Jun 3, 2026

Copy link
Copy Markdown
Member

cc @ArthurZucker

ArthurZucker
ArthurZucker reviewed Jun 4, 2026
View reviewed changes

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed offline! 🤗

Comment thread src/transformers/models/glm_moe_dsa/configuration_glm_moe_dsa.py Outdated
Dovis01 added 3 commits June 4, 2026 11:48
@Dovis01
Remove redundant fields
aa8928c 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Fix tests
45f7dcd 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Remove portentially codepath
7da480e 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
ArthurZucker
ArthurZucker approved these changes Jun 5, 2026
View reviewed changes
Comment thread src/transformers/models/glm_moe_dsa/configuration_glm_moe_dsa.py Outdated
Comment thread src/transformers/models/glm_moe_dsa/modeling_glm_moe_dsa.py
Comment thread utils/check_config_attributes.py Outdated
Dovis01 added 2 commits June 5, 2026 08:29
@Dovis01
Optimize code
f225757 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
@Dovis01
Merge branch 'main' into fix-glm-moe-dsa
d066cd3
@github-actions

github-actions Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: glm_moe_dsa

ArthurZucker
ArthurZucker approved these changes Jun 8, 2026
View reviewed changes

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ty!
We discussed offline can’t use rotation one weights as this would break nope/rope that follows

@ArthurZucker
ArthurZucker merged commit 5161637 into huggingface:main Jun 8, 2026
21 checks passed
@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Jun 8, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

khushali9 pushed a commit to khushali9/transformers that referenced this pull request Jun 8, 2026
@Dovis01 @khushali9
Fix: interleaved RoPE application for MLA and Support Index Cache DSA…
d81bac6 
… indexer skip-topk sharing for GLM5 (huggingface#46372)

* Fix error attributes mapping

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix: Support interleaved RoPE for MLA

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove redundant fields

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove portentially codepath

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Optimize code

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

---------

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
louzongzhi pushed a commit to louzongzhi/transformers that referenced this pull request Jun 10, 2026
@Dovis01 @louzongzhi
Fix: interleaved RoPE application for MLA and Support Index Cache DSA…
4fe20c5 
… indexer skip-topk sharing for GLM5 (huggingface#46372)

* Fix error attributes mapping

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix: Support interleaved RoPE for MLA

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove redundant fields

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove portentially codepath

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Optimize code

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

---------

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
louzongzhi pushed a commit to louzongzhi/transformers that referenced this pull request Jun 10, 2026
@Dovis01 @louzongzhi
Fix: interleaved RoPE application for MLA and Support Index Cache DSA…
45f9a72 
… indexer skip-topk sharing for GLM5 (huggingface#46372)

* Fix error attributes mapping

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix: Support interleaved RoPE for MLA

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove redundant fields

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove portentially codepath

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Optimize code

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

---------

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
This was referenced Jun 18, 2026
Open
Open
@pcuenca pcuenca mentioned this pull request Jun 23, 2026
Merged
6 tasks
@JonathanBerhe JonathanBerhe mentioned this pull request Jul 2, 2026
Merged
stevhliu pushed a commit to stevhliu/transformers that referenced this pull request Jul 30, 2026
@Dovis01 @stevhliu
Fix: interleaved RoPE application for MLA and Support Index Cache DSA…
99b68c5 
… indexer skip-topk sharing for GLM5 (huggingface#46372)

* Fix error attributes mapping

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix: Support interleaved RoPE for MLA

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove redundant fields

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove portentially codepath

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Optimize code

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

---------

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ArthurZucker ArthurZucker ArthurZucker approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Dovis01 @Rocketknight1 @HuggingFaceDocBuilderDev @ArthurZucker