Conversation
This was referenced Jun 19, 2026
yushengsu-thu
added a commit
to radixark/Megatron-Bridge
that referenced
this pull request
Jun 21, 2026
…gatron) megatron-core's DSAIndexer applies RoPE to the LAST qk_pos_emb_head_dim of each index head (split([D-rope, rope])), but the HF/DeepSeek (glm_moe_dsa) checkpoint stores the rope dims in the FIRST half. The previous name->name mapping loaded wq_b/wk/k_norm unchanged, so the indexer rotated the wrong dimensions -- its index scores were structurally decorrelated from the reference (HF<->bridge pearson ~0.48 vs slime ~0.70), flipping ~50% of the sparse top-k key selection at long context. Fix: _IndexerRopeHalfSwapMapping swaps the two halves of each index head's dsa_indexer_head_dim when loading the indexer wq_b, wk, k_norm.weight and k_norm.bias (self-inverse on export); weights_proj (per-head scalar) is untouched. k_norm is included because it is applied to the (swapped) key BEFORE RoPE, so its per-dim scale/bias must be swapped consistently. Mirrors the slime mbridge reference (THUDM/slime#2093 slime_plugins/mbridge/deepseek_v32.py: "training uses last half for rope while DeepSeek uses first half"). Covers GLM-5.1 (index_topk_freq=1) and GLM-5.2 (cross-layer). Validated against slime: all 5 indexer weights become bit-identical, indexer score pearson(slime,bridge) -> 0.98-1.0, and a 3-way HF-reference check shows the fixed bridge tracks HF identically to slime. Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
yushengsu-thu
added a commit
to yushengsu-thu/miles
that referenced
this pull request
Jun 26, 2026
… cuda-graph hang) Migrated from radixark#1374 (THUDM/slime#2093). DeepEP/NVSHMEM's internal NCCL opens a second NCCL communicator that collides with miles' NCCL and hangs during SGLang CUDA-graph replay. Applied to test whether it also resolves the GLM-5.2 dp-attention EVAL-ON deadlock (skeptical: that hang is in CPU/gloo collectives + GLM-5 LoRA uses triton MoE not DeepEP, so NVSHMEM may be inert here -- but empirical test pending a devbox). If it does not fix it, fall back to the miles-side eval grouped-submission fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MuN5ow5hHFxPze36uhgZC4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.