Tiny refactor branching in getting dataset samples - #27
Merged
Conversation
fzyzcjy
changed the title
fzyzcjy
changed the title
fzyzcjy
added a commit
that referenced
this pull request
Mar 19, 2026
JD-ETH
pushed a commit
to JensenFire/miles
that referenced
this pull request
Apr 11, 2026
GuanxingLu
pushed a commit
to GuanxingLu/miles
that referenced
this pull request
Apr 21, 2026
yushengsu-thu
added a commit
that referenced
this pull request
Aug 13, 2026
… grad norm Found by the fixed-bridge GPU re-validation (4xH200 GPT-OSS 20B expert-only LoRA, TP=2+SP, EP=1/ETP=1, full recompute): every reported per-slot grad_norm came out exactly sqrt(2)x the true gradient norm (diag: reduced norm 247.1967 vs local l2 174.7945, ratio 1.41421 on every optim of every slot), and grad_clip_norm under-scaled by the same factor (post-clip l2 0.7071 for clip=1.0). Mechanism: the bridge's grouped-expert adapter weights carry tensor_model_parallel=True unconditionally (upstream-ported attribute stamping in radixark/Megatron-Bridge#27). The only supported multi-LoRA MoE config is expert_tensor_parallel_size=1, where those weights are fully TP-DUPLICATED whenever TP>1 — so Megatron's attribute-based TP-duplicate filter admits every rank's identical gradient into the world-reduced norm and over-counts each logical parameter TP times. This was unobservable before the bridge fix only because rank1's expert-adapter gradients were identically zero (the expert-DDP routing bug); once #27 made them real, the double-count became real too. Fix at the existing pre-wrap seam: after the LoRA transform, clear tensor_model_parallel on grouped-expert adapter weights when TP > expert-TP (the duplicated case), so the stock filter counts each logical param once (TP rank 0) — semantically identical to run-E of the 0812 matrix, which was GPU-verified with true norms and synced ranks. DDP expert-bucket routing keys on 'allreduce' and is untouched; the hook runs pre-wrap so the fp32 masters copy the corrected attribute at optimizer build. Genuinely TP-sharded (attention) adapters keep their flag. CPU regression tests pin the cleared/kept/no-op shapes.
This was referenced Aug 13, 2026
yushengsu-thu
added a commit
that referenced
this pull request
Aug 13, 2026
bridge is now radixark/Megatron-Bridge's default branch and the deliberate integration line for the multi-LoRA work (#27 and follow-ups land there), so the image should follow it without a Dockerfile edit per merge. 005135d had pinned the then-HEAD 688d34b85 by SHA; 76ac997 already returned Dockerfile.rocm to the @bridge form — this aligns the cu13 Dockerfile. The caveat the SHA pin protected against is now stated instead of engineered around: buildkit caches the install layer on the instruction text alone, so a rebuild only picks up new bridge commits with --no-cache (or an explicit cache-bust) — a stale cache silently keeps the old revision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.