ci: Bump MBridge to ec835530efeff71b55ec015f36bcc0c33b8b52b7. by cspades · Pull Request #4363 · NVIDIA-NeMo/RL

cspades · 2026-09-30T16:31:26Z

What does this PR do ?

Bump MBridge to ec835530efeff71b55ec015f36bcc0c33b8b52b7.
Fix some tests that are failing but not directly caused by the bump:
- save_data_plane: true in the two SC recipes (nanov3 token_capture, ppo qwen2.5), needed by the guard from feat(sc): recover replay buffer from native TQ checkpoints #3480 (setup.py).
- The audio padding, needed for vLLM 0.29, which landed in build: bump vllm to 0.29.0 #4080.
- The DPO accuracy threshold.
- The Dynamo fix.

Issues

List issues that this PR closes (syntax):

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

copy-pr-bot · 2026-09-30T16:31:30Z

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

cspades · 2026-09-30T16:32:29Z

/ok to test f9c5536

github-actions · 2026-09-30T16:32:57Z

✅ Submodule Fast-Forward Check Results

Check based on commit: f9c5536 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-09-30T16:40:56Z

/ok to test f0b3eb1

github-actions · 2026-09-30T16:41:29Z

✅ Submodule Fast-Forward Check Results

Check based on commit: f0b3eb1 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-09-30T19:00:30Z

Nightly: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/70750103

github-actions · 2026-09-30T20:25:09Z

✅ Submodule Fast-Forward Check Results

Check based on commit: bde27e9 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-09-30T20:25:14Z

/ok to test 87cc7c8

github-actions · 2026-09-30T20:25:25Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 87cc7c8 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-09-30T20:26:38Z

Another nightly: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/70764248

cspades · 2026-10-01T20:33:36Z

/ok to test 33b4ea0

github-actions · 2026-10-01T20:40:14Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 33b4ea0 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-10-02T15:46:42Z

Previous nightly died and corrupted to HSG nemorl disk quota issues.

Another one: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/71159058

github-actions · 2026-10-03T00:23:00Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 180cf61 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-10-03T00:41:29Z

/ok to test 180cf61

cspades · 2026-10-03T01:06:29Z

Nightly rerun again after the file issues: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/71254681

github-actions · 2026-10-04T03:11:37Z

✅ Submodule Fast-Forward Check Results

Check based on commit: d9caff1 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-10-04T03:39:53Z

✅ Submodule Fast-Forward Check Results

Check based on commit: da421ea (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-10-04T03:43:26Z

Nightly is in a very bad state but one more attempt: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/71428772

cspades · 2026-10-05T16:15:13Z

/ok to test 91188dc

github-actions · 2026-10-05T16:16:17Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 91188dc (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

cspades · 2026-10-05T18:19:06Z

WandB runs are in progress:

https://forge.coreweave.com/wandb/nvidia/nemo-rl/workspace?nw=nwusercye_nv
- grpo-llama3.1-8b-instruct-1n8g-megatron-fp8-rollouts.v3 - This PR.
  - vs. main: https://forge.coreweave.com/wandb/nvidia/nemo-rl/runs/98ub5sv0?nw=nwusercye_nv
- grpo-qwen3-30ba3b-8n8g-megatron-cp2-r3 - This PR.
  - vs. main: https://forge.coreweave.com/wandb/nvidia/nemo-rl/runs/pnfm7rsd?nw=nwusercye_nv

Signed-off-by: Cory Ye <cye@nvidia.com>

…egration. Signed-off-by: Cory Ye <cye@nvidia.com>

Signed-off-by: Cory Ye <cye@nvidia.com>

…andles. Signed-off-by: Cory Ye <cye@nvidia.com>

…alues, and disable vLLM egregious failure from recent bumps. Signed-off-by: Cory Ye <cye@nvidia.com>

Signed-off-by: Cory Ye <cye@nvidia.com>

…cheduler.num_steps on load. Signed-off-by: Cory Ye <cye@nvidia.com>

cspades · 2026-10-06T06:07:47Z

/ok to test 43df240

github-actions · 2026-10-06T06:08:54Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 43df240 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Cory Ye <cye@nvidia.com>

github-actions · 2026-10-06T07:35:08Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 2f53576 (PR #4363 from cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

terrykong · 2026-10-06T07:52:42Z

/ok to test 2f53576

#4695) Signed-off-by: Yuekai Zhang <yuekaiz@nvidia.com> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

cspades self-assigned this Sep 30, 2026

cspades added the CI:L1 Run doctests, unit tests, and functional tests label Sep 30, 2026

cspades requested review from kajalj22 and terrykong September 30, 2026 16:34

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from f9c5536 to f0b3eb1 Compare September 30, 2026 16:39

cspades changed the title ~~Bump MBridge to ec835530efeff71b55ec015f36bcc0c33b8b52b7.~~ ci: Bump MBridge to ec835530efeff71b55ec015f36bcc0c33b8b52b7. Sep 30, 2026

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from bde27e9 to 87cc7c8 Compare September 30, 2026 20:24

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from 33b4ea0 to 180cf61 Compare October 3, 2026 00:21

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from 180cf61 to d9caff1 Compare October 4, 2026 03:10

cspades commented Oct 4, 2026

View reviewed changes

Comment thread nemo_rl/models/generation/megatron/megatron_generation.py

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from da421ea to 5dc0f9f Compare October 4, 2026 03:49

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from 873c9a4 to 91188dc Compare October 5, 2026 16:14

cuichenx mentioned this pull request Oct 5, 2026

feat(ckpt): add a native resume metadata migration hook NVIDIA-NeMo/Megatron-Bridge#6321

Draft

3 tasks done

cspades added 9 commits October 5, 2026 23:07

Bump MBridge to ec835530efeff71b55ec015f36bcc0c33b8b52b7.

9ed8c3a

Signed-off-by: Cory Ye <cye@nvidia.com>

Fix L0.

5d6186a

Signed-off-by: Cory Ye <cye@nvidia.com>

Fix bug introduced in #3920 that inadvertently removed Dynamo Gym int…

a13b838

…egration. Signed-off-by: Cory Ye <cye@nvidia.com>

Fix configs after MBridge bump.

235d777

Signed-off-by: Cory Ye <cye@nvidia.com>

Fix Ray validation guard for actors that mock dependencies of their h…

ab82940

…andles. Signed-off-by: Cory Ye <cye@nvidia.com>

Fix vLLM 0.29 audio padding, minor update to failing nightly golden v…

c8f8770

…alues, and disable vLLM egregious failure from recent bumps. Signed-off-by: Cory Ye <cye@nvidia.com>

Rebase and revert specific nightly fixes for test authors.

9feb779

Signed-off-by: Cory Ye <cye@nvidia.com>

Address review comments.

4e87fd5

Signed-off-by: Cory Ye <cye@nvidia.com>

Checkpoint train_state.consumed_train_samples, and override it with s…

43df240

…cheduler.num_steps on load. Signed-off-by: Cory Ye <cye@nvidia.com>

cspades force-pushed the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch from 91188dc to 43df240 Compare October 6, 2026 06:07

Add alarm for load_checkpoint hook arg.

2f53576

Signed-off-by: Cory Ye <cye@nvidia.com>

terrykong approved these changes Oct 6, 2026

View reviewed changes

terrykong enabled auto-merge (squash) October 6, 2026 16:46

terrykong merged commit 8661b47 into main Oct 6, 2026
234 of 240 checks passed

terrykong deleted the cye/bump-mbridge-ec835530efeff71b55ec015f36bcc0c33b8b52b7 branch October 6, 2026 21:42

lauradang mentioned this pull request Oct 7, 2026

feat(token-capture): carry media through Megatron capture rollouts #4332

Merged

yuki-97 mentioned this pull request Oct 8, 2026

fix(recipes): enable save_data_plane for GRPO token-capture SC nightly #4587

Merged

bg51717 mentioned this pull request Oct 12, 2026

test(ppo): check that consecutive async checkpoints write fresh optimizer state #4329

Merged

yuki-97 mentioned this pull request Oct 13, 2026

fix(ci): fix consistently failing and flaky nightly test gates #4689

Merged

yuekaizhang mentioned this pull request Oct 13, 2026

fix: pad audio to hop length for Qwen3-Omni prompt match (cp of #4363) #4695

Merged

2 of 4 tasks

yuki-97 pushed a commit that referenced this pull request Oct 13, 2026

fix: pad audio to hop length for Qwen3-Omni prompt match (cp of #4363) (

19666f8

#4695) Signed-off-by: Yuekai Zhang <yuekaiz@nvidia.com> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

Conversation

cspades commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Sep 30, 2026

Uh oh!

cspades commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Sep 30, 2026

Uh oh!

cspades commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Oct 3, 2026

Uh oh!

cspades commented Oct 3, 2026

Uh oh!

github-actions Bot commented Oct 4, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

Uh oh!

github-actions Bot commented Oct 4, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Oct 4, 2026

Uh oh!

cspades commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cspades commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

cspades commented Sep 30, 2026 •

edited

Loading

cspades commented Oct 2, 2026 •

edited

Loading

cspades commented Oct 5, 2026 •

edited

Loading