Conversation
dion's Muon distributes orthogonalization by sharding a parameter's leading dim
across the distributed mesh and asserts divisibility ('Shard dimension 0 size 72
is not divisible by world size 16'). Models with small projections whose leading
dim does not divide the mesh (e.g. attention-gate weights of shape (72, hidden))
crashed on the first optimizer step.
Extend the muon_enabled predicate to route such parameters to the existing adamw
group, like the other muon-unsuitable parameters (1D, embeddings, lm_head). The
check uses the same mesh the default muon group is constructed with; expert
groups on a dedicated EP mesh are unaffected in practice since fused expert dims
divide cleanly.
Fixes #3437.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hallerite
marked this pull request as ready for review
August 30, 2026 19:56
hallerite
marked this pull request as draft
August 30, 2026 20:18
Collaborator
|
This would silently introduce different optimization based on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3437: the first Muon step crashes training a large custom-impl MoE checkpoint on a 2-train-node topology:
Root cause (corrected after review — this is not a #3411 regression, and nothing changed in how weights reach the optimizer): dion's distributed Muon shards each parameter's leading dim across the mesh and asserts divisibility. The checkpoint has heterogeneous attention gating — 12 full-attention layers with
g_proj (48, hidden)and 36 sliding-attention layers withg_proj (72, hidden). The muon mesh isdp_shard_cp: 72 divides a 1-node world (8) but not a 2-node world with cp=4 (16) — so the same config steps fine on one topology and asserts on another. Dion pin and the muon param-grouping are unchanged for months; the failure is a divisibility accident of topology × checkpoint.Fix: extend the
muon_enabledpredicate — parameters whose dim0 is not divisible by the muon mesh size go to the existing adamw group, exactly like the other muon-unsuitable parameters (1D, embeddings,lm_head). That makes routing topology-robust; for the affected model it's 36 small gate matrices (~8M params), negligible for optimization.Notes:
dp_shard_cp, else world). The expert group's dedicated EP mesh can differ; fused expert dims divide cleanly in practice — a per-group check is the follow-up if that ever changes.🤖 Generated with Claude Code