Skip to content

chore(beep boop 馃): Bump uv.lock (main, mcore-main) (2026-06-04) - #4153

Closed
svcnvidia-nemo-ci wants to merge 1 commit into
mainfrom
bump-ci-container-2026-06-04-main-main
Closed

chore(beep boop 馃): Bump uv.lock (main, mcore-main) (2026-06-04)#4153
svcnvidia-nemo-ci wants to merge 1 commit into
mainfrom
bump-ci-container-2026-06-04-main-main

Conversation

@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

馃殌 PR to bump uv.lock in main.

馃 This PR will be merged automatically once CI passes.

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor Author

/ok to test c005842

@copy-pr-bot

copy-pr-bot Bot commented Jun 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33

yaoyu-33 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

MCore bump auto-fix status for main:

Classification: MCore broke Bridge
Evidence: CICD NeMo run 26946759697 failed on 2026-06-04 after the main / mcore-main bump advanced 3rdparty/Megatron-LM from 64c3fb86ca2b1b875c0206500195b4de2977592a to 471e3e38537869f332d9c97942198693009e15d8. Representative failed jobs L0_Launch_recipes_llama_1b (79513077752), L2_Launch_ckpts_mbridge_to_mlm_qwen3_4b (79513077417), and L0_Launch_models_deepseek (79513077719) all hit model_provider.finalize() and then 3rdparty/Megatron-LM/megatron/core/model_parallel_config.py:437 with AssertionError: Transformer Engine cross entropy loss fusion is disabled due to stability issues. Use cross_entropy_fusion_impl='native', or disable cross_entropy_loss_fusion. The MCore compare includes 168cb15d70ad4a3ca9e3b3e396adc6201a6f50ff from 2026-06-03, Disable TE cross entropy loss fusion (#5115), which added that assertion.
Fix PR: not opened. A previous relevant fix PR exists but was closed unmerged: #4138 (fix(recipes): default cross entropy fusion to native), closed on 2026-06-03 at 10:26 PDT. It appears to cover this same root cause by switching Bridge recipe/provider/perf defaults from cross_entropy_fusion_impl="te" to "native". Per the auto-fix guardrail, I am not opening a replacement until maintainers confirm whether #4138 should be revived/reopened or replaced.
Guards: none added or removed.
Validation: no new branch or fix validation run. Read Linear MB-491, PR metadata/diff/checks/comments, Actions job logs via the job-log API, and the MCore compare/commit patch. Local/CW tests were not run because no replacement fix was created.
Next action: maintainer decision needed: reopen/revive #4138, approve a replacement fix PR, or choose the alternate policy of disabling cross_entropy_loss_fusion instead of switching to native CE fusion.

Handoff time: 2026-06-04 07:03 PDT.

@yaoyu-33 yaoyu-33 closed this Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants