fix(te): pass pad_between_seqs explicitly for tail-padded THD under CP

[JackRao123]

The bug

Packed (THD) batches describe sequence boundaries with two lists of running totals:

• cu_seqlens — counting real tokens only. One 698-token sequence → [0, 698].
• cu_seqlens_padded — including padding. Padded to 704 → [0, 704].
• pad_between_seqs — a flag telling TE the two lists differ. True → TE computes each CP rank's token counts exactly. False → shortcut: divide the totals by the number of ranks.

TE guesses the flag by comparing the lists without their last entry ([:-1]). Padding at the end of the batch only shows in the last entry — so for our batches TE always guesses False and takes the shortcut, which miscounts whenever the real length doesn't divide evenly:

13 real tokens (t0–t12), padded to 16. CP=2, zigzag chunks of 4:
rank 0 holds chunks 0+3, rank 1 holds chunks 1+2.

           rank 0                        rank 1
tokens:  t0 t1 t2 t3 | t12 p13 p14 p15   t4 t5 t6 t7 | t8 t9 t10 t11
truth:   5 of 8 real                     8 of 8 real

TE's guess (pad_between_seqs=False):  13 // 2 = 6 real, on both ranks.

  rank 0: p13 (padding) counted as real → reads uninitialized memory
          → different answer every run
  rank 1: t10, t11 (real) counted as padding → dropped from attention
          → silently wrong, same every run

The first failure mode is what broke the Nemotron nightly ckpt-roundtrip gate; the second ships silently in every CP>1 THD training run.

The fix

Don't let TE guess: we have both lists at the call site, so compare them fully (last entry included) and pass the flag explicitly. Scoped to CP because that's the only place the shortcut is wrong — setting the flag elsewhere would needlessly switch attention kernels.

Validation

• Standalone 4-GPU repro: bitwise-deterministic fwd+bwd, 100/100 runs × 3 seeds × CP{2,4}; CP4 matches CP1 ground truth to bf16 rounding.
• Full trainer stack (TP2×CP4): 10/10 bit-identical forwards vs a nondeterministic control.
• Full record: trainers repo, experiment_artefacts/lps1073/ (LPS-1073, split from the original LPS-1063).

Downstream bumps: basetenlabs/Megatron-Bridge#31 → basetenlabs/trainers#994.

🤖 Generated with Claude Code

[linear]

LPS-1063

[jerryhong21]

lets clean up these comments # slop