Add GLM-5 model support (glm_moe_dsa) - #881
Closed
iamadalek wants to merge 1 commit into
Closed
Conversation
4 tasks
Combines DeepSeek V3.2 DSA sparse attention indexer with GLM-4 MoE Lite MLA attention and PipelineMixin. Key design decisions: - Dual cache (CacheList) with CacheList-aware pipeline integration - Both MLA and Indexer RoPE use traditional=True (interleaved) - rope_parameters nested config extraction via __post_init__ - No FP8 dequantization (BF16 model) - Indexer stays unsharded in distributed mode Closes ml-explore#879 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
iamadalek
force-pushed
the
feature/glm-moe-dsa-879
branch
from
February 12, 2026 02:55
01965ad to
bd732d6
Compare
Author
Contributor
|
A PR already exists #867 |
Author
A apologies - I missed that. Thank you for doing it. |
Contributor
|
@iamadalek no worries, happens to the best of us. cheers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mlx_lm/models/glm_moe_dsa.pyfor the GLM-5 (glm_moe_dsa) architecturetraditional=True(interleaved)rope_parametersnested config via__post_init__test_all_modelswithindex_topk=2to exercise the sparse attention pathCloses #879
Test plan
python -m unittest discover tests/— all 157 tests passpre-commit run --files— black + isort clean🤖 Generated with Claude Code