ci: Bump Megatron-Bridge to b11414c7 - #3906
Merged
Merged
Conversation
Contributor
Author
|
/ok to test ea95f20 |
Contributor
|
/ok to test aab7ed1 |
Contributor
|
/ok to test 167e1c0 |
cspades
force-pushed
the
kajalj/bump-emerging-optimizers-v0.3.0
branch
from
August 31, 2026 15:39
167e1c0 to
72211f4
Compare
github-actions
Bot
added
the
Documentation
Contributor
|
/ok to test 72211f4 |
Bumps the Megatron-Bridge submodule (nesting Megatron-LM at 7c9c3a027, which pins emerging-optimizers@v0.3.0) and RL's own constraint-dependencies pin to match, per <NVIDIA/Megatron-LM#6701>. The automated mbridge-bump pipeline (nemo-ci) could not do this on its own: it bumps only the submodule, not RL's root pin, so uv lock failed with a conflicting-URL error until both moved together in one commit. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
cspades
force-pushed
the
kajalj/bump-emerging-optimizers-v0.3.0
branch
from
August 31, 2026 15:43
72211f4 to
75f7f04
Compare
|
Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
/ok to test 75f7f04 |
…optimizers-v0.3.0 Signed-off-by: Kajal Jain <kajalj@nvidia.com> # Conflicts: # uv.lock
Contributor
|
/ok to test 8fc0e3d |
yuki-97
approved these changes
Sep 1, 2026
yuki-97
enabled auto-merge (squash)
September 1, 2026 07:21
yuki-97
added
CI:Lfast
Contributor
|
/ok to test 807e12a |
Contributor
|
8fc0e3d passed on |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
b11414c7(nesting Megatron-LM at7c9c3a027).emerging-optimizerspin inconstraint-dependenciesfromv0.2.0tov0.3.0, matching Megatron-LM's[tool.uv.sources]pin (see NVIDIA/Megatron-LM#6701).uv.lock.Manual bump: the automated
create_mbridge_bump_prnemo-ci pipeline only advances the submodule pointer, not RL's own root pin, souv lockfailed with conflictingemerging-optimizersURLs (v0.2.0from RL's stale root pin vsv0.3.0transitively via the bumped Megatron-LM). Bumping both together in this PR resolves cleanly.Supersedes #3868 (stale automated bump to an older commit).
Potential Open Issues (Related to MBridge)
Summary of the TODO's and relevant issues:
infissue will be fixed in a future bump: Avoid emitting NaN / Inf logprobs out of inference NVIDIA/Megatron-LM#69935.13or applying this hot-fix: fix: handle symlinked Transformers module cache #3914Codex summary of all nightly test failures that may need tracking:
417124452 — Moonlight TQ
Test:
grpo-moonlight-16ba3b-4n8g-megatron-tq_simple.shCheckpointException: checkpoint missing expectedlinear_q_proj.layer_norm_weight.417124388 — Moonlight
Test:
grpo-moonlight-16ba3b-4n8g-megatron.shSame checkpoint/state-dict mismatch as above.
417124464 — Qwen2.5 native Megatron generation
Test:
grpo-qwen2.5-math-1.5b-instruct-1n8g-megatron_generation-noncolocated-single-controller-sync.shtrain/token_mult_prob_error = inf, expected median< 1.1.The two Moonlight failures are one issue: regenerate/invalidate the shared checkpoint after the MBridge model-schema change. The generation failure needs a pre/post-bump A/B run.
417124437 — Nemotron3 Super
Test:
grpo-nemotron3-super-120BA12B-16n8g-megatron.shCUDA illegal memory access in
offload_after_refit()after accessing offloaded Mamba weights.417124430 — Nano-v3 Megatron
Test:
grpo-nanov3-30BA3B-2n8g-megatron-pack-cp.shSame Mamba offload CUDA illegal-memory access.
417124425 — Nano-v2 Megatron
Test:
grpo-nano-v2-12b-1n8g-megatron.shSame Mamba offload CUDA illegal-memory access.
417124401 — Nemotron-Omni MMPR Megatron
Test:
vlm_grpo-nemotron-omni-30ba3b-mmpr-4n8g-megatron-tp8ep16.v1.shFileNotFoundError: .../blobs/configuration_nemotron_h.py.417124400 — Nemotron-Omni CLEVR Megatron
Test:
vlm_grpo-nemotron-omni-30ba3b-clevr-1n8g-megatron-tp8ep8.v1.shSame Transformers 5.12.1 symlink-resolution error.
The IMA group is covered by Peter Dykas’s fix; the two multimodal jobs are covered by Transformers ≥5.13.
417124419
Test:
grpo-dapomath17k-dsv3-megatron.shDeepseekV3Configlacksqk_head_dim. Identical in prior job 415927951.417124408
Test:
grpo-moonlight-16ba3b-4n8g-megatron-fp8-e2e.shvLLM CuMem allocator
CUDA invalid argument. Identical in prior job 415927940.417124397
Test:
vlm_grpo-qwen3-omni-30ba3b-audiomcq-4n8g-megatron.v1.shGloo receive timed out after 30 minutes.
417124396
Test:
vlm_grpo-qwen2.5-omni-7b-intent-1n8g-megatron.v1.shReached the Slurm time limit without a preceding exception.
Test plan
uv lockresolves cleanly (565 packages)uv.lockretainsprovides-extrasmetadata