Skip to content

cp: Disable TE cross entropy loss fusion (5115) into core_r0.18.0 - #5144

Merged
ko3n1g merged 1 commit into
core_r0.18.0from
cherry-pick-5115-core_r0.18.0
Jun 25, 2026
Merged

cp: Disable TE cross entropy loss fusion (5115) into core_r0.18.0#5144
ko3n1g merged 1 commit into
core_r0.18.0from
cherry-pick-5115-core_r0.18.0

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

beep boop [🤖]: Hi @mchrzanowski 👋,

we've cherry picked #5115 into  for you! 🚀

Please review and approve this cherry pick by your convenience!

Co-authored-by: Mike Chrzanowski <mchrzanowski@gcp-nrt-cs-001-login-001.cm.cluster>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
@ko3n1g
ko3n1g requested review from a team as code owners June 3, 2026 23:11
@ko3n1g
ko3n1g requested a review from mchrzanowski June 3, 2026 23:11
@ko3n1g

ko3n1g commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test ff64144

@copy-pr-bot

copy-pr-bot Bot commented Jun 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ko3n1g
ko3n1g marked this pull request as draft June 4, 2026 06:19
@ko3n1g

ko3n1g commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Deferred to the 0.18.1 release due to the large diff. Release testing for both convergence and performance is already ongoing and shouldn't be replaced anymore at this point.

@ko3n1g
ko3n1g marked this pull request as ready for review June 25, 2026 09:28
@ko3n1g
ko3n1g merged commit 078344e into core_r0.18.0 Jun 25, 2026
83 of 85 checks passed
@ko3n1g
ko3n1g deleted the cherry-pick-5115-core_r0.18.0 branch June 25, 2026 09:28
@zpguo23

zpguo23 commented Jul 30, 2026

Copy link
Copy Markdown

Hi @mchrzanowski , I’m investigating why TE cross-entropy loss fusion was disabled in the original PR #5115 and cherry-picked into this branch. The original PR mentions “observed training stability issues,” but does not describe the specific failure mode or link a related issue.

I checked TransformerEngine issue #2734 and PR #2746. That bug could silently produce incorrect loss values for non-contiguous transposed inputs, but the TE commit pinned by this Megatron branch (4220403e831d29e93868f7793693ea83f6b8b05b) already contains that fix.

Could you please clarify what specific issue motivated the disablement? For example, was it loss divergence, NaNs, incorrect gradients, or a problem under a particular model configuration or TE version? Is there a related issue or reproducer that can be shared? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants