Bump TransformerEngine to 2.17.0 - #1781
Merged
Merged
Conversation
- transformer_engine / transformer_engine_cu13 / transformer_engine_torch 2.12.0 -> 2.17.0 - update te_fa2_sm103_whitelist.patch for TE 2.17 source layout - add te_dequantized_backward_override.patch (hot fix from NVIDIA/TransformerEngine#3141; drop after TE v2.18)
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Zhichenzzz
approved these changes
Jul 24, 2026
Zhichenzzz
added a commit
that referenced
this pull request
Jul 24, 2026
The wheels release still ships transformer_engine_torch 2.12.0; installing it after the 2.17.0 bump (#1781) downgrades transformer-engine-cu13 via its dependency pin and the TE import sanity check fails the build. TE now comes from PyPI, so remove the wheel branch (and delete the stale wheel from /tmp/wheels so nothing else can pick it up).
yueming-yuan
added a commit
that referenced
this pull request
Jul 26, 2026
…ge#24 Validates the TE 2.17 grouped-linear contract fix end to end. The `not enough values to unpack (expected 21, got 16)` failures in the test_glm5_*_lora_ci tests come from megatron-bridge's `_forward_te_grouped_linear` calling TE's private `_GroupedLinear` with TE <=2.14's positional layout; miles picked that up when it bumped TransformerEngine 2.12 -> 2.17 in #1781. Revert to `@bridge` -- ideally pinned to the merged commit rather than a branch, as line 82 does for mbridge -- once #24 lands. Note this does not address `window_size_left`, which is a separate TE 2.17 vs flash-attn skew inside TE's own context-parallel path.
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps TransformerEngine 2.12.0 → 2.17.0 (cu13 trio:
transformer_engine/transformer_engine_cu13/transformer_engine_torch; cu12 path 2.10.0 → 2.17.0). This is the dependency prerequisite for #1261 (NVFP4 RL), which constructstransformer_engine.pytorch.tensor.nvfp4_tensor.NVFP4Quantizerdirectly.TE patches
te_dequantized_backward_override.patch(new): hot fix from [PyTorch] Preserve fprop operands for dequantized backward override NVIDIA/TransformerEngine#3141; drop after TE v2.18.te_fa2_sm103_whitelist.patch: updated for the TE 2.17 source layout.Both patch files are regenerated from pristine pip-installed TE 2.17.0 sources so that they apply cleanly with both GNU
patch -p1(fuzz 0) andgit apply. The originals from the #1261 branch fail the Dockerfile'spatch -p1step (the dequantized patch is rejected by GNU patch 2.7.6 even at default fuzz; the fa2 patch needs fuzz 2 and is rejected bygit applyas corrupt), which would break the image build.Validation
tests/fast-gpu/test_nvfp4_quantizer.py: 633 passed on a B200 devbox with TE 2.17.0 + both patches.--rollout-nvfp4 --train-nvfp4): NVFP4 checkpoint conversion, sglangflashinfer_trtllm_routedrollout, and bf16+fp4 Megatron training all run; steps 0-2 stable withtrain_rollout_logprob_abs_diff≈ 0.054-0.056 andtrain_rollout_kl≈ 0.012-0.013 (same order as the step-1 sanity numbers reported in NVFP4 RL #1261).te.Linearfwd/bwd and rowwiseNVFP4Quantizersmoke-tested on GPU after upgrade.Note for #1261: after this lands, rebase onto main and drop the branch's copies of the Dockerfile TE bump and the two patch files (this PR's regenerated patch bytes must win, or the docker build fails at the patch step).