fix(mtp): use padded cu_seqlens in MTP roll for THD with CP - #4495
Open
BestJuly wants to merge 2 commits into
Open
fix(mtp): use padded cu_seqlens in MTP roll for THD with CP#4495BestJuly wants to merge 2 commits into
BestJuly wants to merge 2 commits into
Conversation
`_roll_tensor_packed_seq` indexed local chunks with the unpadded `cu_seqlens_q`, but with CP the THD layout is produced by `tex.thd_get_partitioned_indices(cu_seqlens_padded, ...)`, which requires each per-sequence padded length to be divisible by 2*cp_size. When real seqlens are not multiples of 2*cp_size (e.g. odd lengths), `// cp_size` gave the wrong local boundaries: `chunk(2)` split unevenly, neighbour sends/recvs had mismatched sizes, and tokens leaked across sequence boundaries (very small seqs hit IndexError). Prefer `cu_seqlens_q_padded` when provided (with fallback to the unpadded version), matching the convention already used in `attention.py`. Also adds a parametrized unit test covering odd seqlens for CP=1 and CP=2 (`test_roll_tensor_with_packed_sequences_odd_seqlen`), with the CP=2 case using padded `[8, 12]` over real `[7, 11]`. Co-Authored-By: Jingliang Li <jinliangl@nvidia.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Victarry
approved these changes
May 12, 2026
This was referenced May 12, 2026
Phlip79
approved these changes
Jun 8, 2026
Contributor
|
/ok to test bc6bace |
FDecaYed
added a commit
to FDecaYed/Megatron-LM
that referenced
this pull request
Jul 21, 2026
Add configurable boundary-fill semantics and roll packed THD MTP inputs across contiguous context-parallel shards without crossing sequence boundaries. Reconstructed from the MTP portion of NVIDIA#5795 on top of the padded-cu-seqlens fix in NVIDIA#4495. Signed-off-by: Deyu Fu <Deyu.Foo@gmail.com>
24 tasks
jaredcasper
approved these changes
Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_roll_tensor_packed_seq(Multi-Token Prediction, THD packed sequences with Context Parallelism).tex.thd_get_partitioned_indices(cu_seqlens_padded, ...)and requires every per-sequence padded length to be divisible by2*cp_size. The roll function indexed local chunks with the unpaddedcu_seqlens_q, so when real seqlens are not multiples of2*cp_size(e.g. odd lengths),// cp_sizeproduced wrong local boundaries:chunk(2)split unevenly, neighbour send/recv tensors had mismatched sizes, and tokens leaked across sequence boundaries (very small seqs hitIndexError).cu_seqlens_q_paddedwhen provided, with fallback tocu_seqlens_q(matches the convention already used inattention.py).test_roll_tensor_with_packed_sequences_odd_seqlencovering odd seqlens for CP=1 and CP=2.Test plan
pytest tests/unit_tests/transformer/test_multi_token_prediction.py::TestMultiTokenPrediction::test_roll_tensor_with_packed_sequences_odd_seqlen— CP=1 PASSED on a single GPU.torchrun --nproc_per_node=2 -m pytest tests/unit_tests/transformer/test_multi_token_prediction.py::TestMultiTokenPrediction::test_roll_tensor_with_packed_sequences_odd_seqlen— both CP=1 and CP=2 PASSED.test_roll_tensor_with_packed_sequences(CP=1 / CP=2) still PASSES, no regression.Reproduction (before fix, CP=2)
With odd seqlens
[7, 11]padded to[8, 12]:With sequences smaller than
2*cp_size(e.g.[3]padded to[4]), the un-fixed code raisesIndexError.Cross-reference
PR for
dev: #4494