Skip to content

[DO NOT MERGE] fix: route muon-indivisible params to adamw (topology-dependent dion assert) - #3438

Closed
hallerite wants to merge 1 commit into
mainfrom
fix/muon-indivisible-dim0
Closed

hallerite wants to merge 1 commit into
mainfrom
fix/muon-indivisible-dim0

Conversation

@hallerite

@hallerite hallerite commented Aug 30, 2026

Copy link
Copy Markdown
Member

Fixes #3437: the first Muon step crashes training a large custom-impl MoE checkpoint on a 2-train-node topology:

dion/muon.py:457 muon_update_batch_async
AssertionError: Shard dimension 0 size 72 is not divisible by world size 16.

Root cause (corrected after review — this is not a #3411 regression, and nothing changed in how weights reach the optimizer): dion's distributed Muon shards each parameter's leading dim across the mesh and asserts divisibility. The checkpoint has heterogeneous attention gating — 12 full-attention layers with g_proj (48, hidden) and 36 sliding-attention layers with g_proj (72, hidden). The muon mesh is dp_shard_cp: 72 divides a 1-node world (8) but not a 2-node world with cp=4 (16) — so the same config steps fine on one topology and asserts on another. Dion pin and the muon param-grouping are unchanged for months; the failure is a divisibility accident of topology × checkpoint.

Fix: extend the muon_enabled predicate — parameters whose dim0 is not divisible by the muon mesh size go to the existing adamw group, exactly like the other muon-unsuitable parameters (1D, embeddings, lm_head). That makes routing topology-robust; for the affected model it's 36 small gate matrices (~8M params), negligible for optimization.

Notes:

  • The check uses the mesh the default muon group is built on (dp_shard_cp, else world). The expert group's dedicated EP mesh can differ; fused expert dims divide cleanly in practice — a per-group check is the follow-up if that ever changes.
  • Alternatives considered: padding/remainder handling inside dion (external package) — heavier, and arguably the better long-term home; this PR keeps prime-rl safe either way.

🤖 Generated with Claude Code

dion's Muon distributes orthogonalization by sharding a parameter's leading dim
across the distributed mesh and asserts divisibility ('Shard dimension 0 size 72
is not divisible by world size 16'). Models with small projections whose leading
dim does not divide the mesh (e.g. attention-gate weights of shape (72, hidden))
crashed on the first optimizer step.

Extend the muon_enabled predicate to route such parameters to the existing adamw
group, like the other muon-unsuitable parameters (1D, embeddings, lm_head). The
check uses the same mesh the default muon group is constructed with; expert
groups on a dedicated EP mesh are unaffected in practice since fused expert dims
divide cleanly.

Fixes #3437.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hallerite
hallerite marked this pull request as ready for review August 30, 2026 19:56
@hallerite hallerite changed the title fix: route muon-indivisible params to adamw instead of crashing the step fix: route muon-indivisible params to adamw (topology-dependent dion assert) Aug 30, 2026
@hallerite
hallerite marked this pull request as draft August 30, 2026 20:18
@S1ro1

S1ro1 commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

This would silently introduce different optimization based on dp_shard_cp size and also route various 2D params silently to adamw even though they should've been optimized by Muon. Should fix in DION library

@S1ro1 S1ro1 closed this Aug 30, 2026
@hallerite hallerite changed the title fix: route muon-indivisible params to adamw (topology-dependent dion assert) [DO NOT MERGE] fix: route muon-indivisible params to adamw (topology-dependent dion assert) Aug 30, 2026
@hallerite
hallerite deleted the fix/muon-indivisible-dim0 branch August 30, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Muon step crashes on custom MoE after #3411 layout unification: shard dim not divisible by world size

2 participants