cp: Disable TE cross entropy loss fusion (5115) into core_r0.18.0 - #5144
Conversation
Co-authored-by: Mike Chrzanowski <mchrzanowski@gcp-nrt-cs-001-login-001.cm.cluster> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
|
/ok to test ff64144 |
|
Deferred to the 0.18.1 release due to the large diff. Release testing for both convergence and performance is already ongoing and shouldn't be replaced anymore at this point. |
|
Hi @mchrzanowski , I’m investigating why TE cross-entropy loss fusion was disabled in the original PR #5115 and cherry-picked into this branch. The original PR mentions “observed training stability issues,” but does not describe the specific failure mode or link a related issue. I checked TransformerEngine issue #2734 and PR #2746. That bug could silently produce incorrect loss values for non-contiguous transposed inputs, but the TE commit pinned by this Megatron branch (4220403e831d29e93868f7793693ea83f6b8b05b) already contains that fix. Could you please clarify what specific issue motivated the disablement? For example, was it loss divergence, NaNs, incorrect gradients, or a problem under a particular model configuration or TE version? Is there a related issue or reproducer that can be shared? Thanks! |
beep boop [🤖]: Hi @mchrzanowski 👋,