Skip to content

chore: bump transformer-engine to 2.14.1 - #2434

Merged
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
zpqiu:codex/bump-transformer-engine-2.14.1
May 8, 2026
Merged

chore: bump transformer-engine to 2.14.1#2434
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
zpqiu:codex/bump-transformer-engine-2.14.1

Conversation

@zpqiu

@zpqiu zpqiu commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Bump TransformerEngine to the v2.14.1 release for both automodel and mcore extras.
  • Keep the CUDA 12 core extra explicit while resolving the git tag through uv.lock.
  • Update uv dependency metadata for the new TE source version.

Related Issues

#2320

Additional Info

The current TE version pinned by RL has a checkpoint resume problem with FusedAdam.

That snapshot includes the FusedAdam DTensor/local-state change, but misses the later DCP compatibility fix. As a result, FusedAdam optimizer states for degenerate sharded params such as shared_expert_gate.weight [1, 2048] were saved with rank-local metadata like (0, 2048) on most ranks and (1, 2048) on a few ranks. Model weights metadata were fine; only optimizer state metadata was corrupted.

The fix is to bump Transformer Engine to a version containing a4a073bbb7c625727b006adacc2a19f4a80cdb61, e.g. TE v2.14 commit f031cf87bd054c7558b887df7bed93975456667f.

Related PR: NVIDIA/TransformerEngine#2795

Signed-off-by: Zhaopeng Qiu <alexq@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented May 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@zpqiu zpqiu changed the title build: bump transformer-engine to 2.14.1 chore: bump transformer-engine to 2.14.1 May 7, 2026
@zpqiu
zpqiu marked this pull request as ready for review May 7, 2026 09:19
@zpqiu
zpqiu requested a review from a team as a code owner May 7, 2026 09:19
@zpqiu zpqiu added the CI:L1 Run doctests, unit tests, and functional tests label May 7, 2026
@zpqiu

zpqiu commented May 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0d05f0a

@zpqiu

zpqiu commented May 8, 2026

Copy link
Copy Markdown
Contributor Author

Hi @terrykong, Sharon has run the nightly tests twice. Here is the CI link: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/50603458. Some tests failed, but they don’t seem to be related to this bump PR. The analysis report generated by CC is below. Do these results look acceptable to you?
image (4)

@terrykong

Copy link
Copy Markdown
Collaborator

Yea, i think those failures seem unrelated, this PR lgtm

@terrykong
terrykong enabled auto-merge (squash) May 8, 2026 04:32
@terrykong
terrykong merged commit f947d78 into NVIDIA-NeMo:main May 8, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants