Skip to content

cp: Fix UT timeout (#4310) - #4373

Merged
ko3n1g merged 2 commits into
NVIDIA:mainfrom
chtruong814:chtruong/cherry-pick-4310
Apr 18, 2026
Merged

cp: Fix UT timeout (#4310)#4373
ko3n1g merged 2 commits into
NVIDIA:mainfrom
chtruong814:chtruong/cherry-pick-4310

Conversation

@chtruong814

@chtruong814 chtruong814 commented Apr 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do ?

cp: Fix UT timeout (#4310) from the dev branch

This was intended to fix the timeout issues but it was merged into dev instead of main. So, cherry-picking this to see if it improves test stability.

#4300

Also, reverting this change to mark one of the tests as flaky #4299

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

kunlunl and others added 2 commits April 17, 2026 21:44
@kunlunl Kunlun Li @chtruong814
Fix UT timeout (NVIDIA#4310)
1075b67 
Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Revert "ci: mark test_fused_indexer_loss_gradient_tp_consistency as f…
5b68b88 
…laky_in_dev (NVIDIA#4299)"

This reverts commit 3d7a701.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft April 18, 2026 02:47
@github-actions

github-actions Bot commented Apr 18, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@copy-pr-bot

copy-pr-bot Bot commented Apr 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@chtruong814
chtruong814 marked this pull request as ready for review April 18, 2026 02:47
@svcnvidia-nemo-ci svcnvidia-nemo-ci added this to the Core 0.16 milestone Apr 18, 2026
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team April 18, 2026 02:47
@ko3n1g
ko3n1g added this pull request to the merge queue Apr 18, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Apr 18, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24598810294

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Apr 18, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24600062781

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 18, 2026
@ko3n1g
ko3n1g added this pull request to the merge queue Apr 18, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Apr 18, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24601224772

Merged via the queue into NVIDIA:main with commit 664baa8 Apr 18, 2026
72 checks passed
Victarry pushed a commit to yanring/Megatron-LM that referenced this pull request Apr 20, 2026
Dennis Liu
Merge remote-tracking branch 'origin/main' into pr3431/fix-conflicts
9d4c1bc 
* origin/main: (286 commits)
  Rename MambaModel/MambaStack to HybridModel/HybridStack (NVIDIA#4099)
  Fix Megatron initialization with extra_args_provider (NVIDIA#4327)
  Fix RL to once again work with --skip-train (NVIDIA#4249)
  Add activation logging and tokens per expert logging (NVIDIA#3842)
  Make param_index_map always use unpacked (full numel) offsets (NVIDIA#4328)
  FA4 Inference (NVIDIA#4186)
  Fix RL reward due to stop token (NVIDIA#4096)
  cp: Fix UT timeout (NVIDIA#4310) (NVIDIA#4373)
  feat(ckpt): add --async-ckpt-use-cpu-shm argument (NVIDIA#4355)
  Update copy-pr-bot.yaml [skip ci]
  Docs: improve docstrings and comments in example training loop (NVIDIA#4041)
  Add QK layernorm support for dot-product attention in MambaModel (NVIDIA#4067)
  Fix bug with non-partial rollouts (NVIDIA#3964)
  [docs] ci: use parent-relative json_url for version picker (NVIDIA#4367)
  Add tables and histogram for RL staleness (NVIDIA#4097)
  Port DeepSeek Sparse Attention to `MambaModel` (NVIDIA#3553)
  docs: bump versions1.json to 0.17.0 (latest) (NVIDIA#4360)
  Fix potential coredump issue that occurs when saving a checkpoint (NVIDIA#1871)
  ci(gb200): add 1-node mr-github functional test variants (NVIDIA#4334)
  fix: wait for async P2P send before deallocating output tensor (NVIDIA#4047)
  ...

# Conflicts:
#	megatron/core/transformer/cuda_graphs.py
yangbofun pushed a commit to xlm-research/Megatron-LM that referenced this pull request May 22, 2026
@chtruong814 @kunlunl Kunlun Li
cp: Fix UT timeout (NVIDIA#4310) (NVIDIA#4373)
cdb9754 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>
yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026
@chtruong814 @kunlunl
Kunlun Li @yhgalaxy
cp: Fix UT timeout (NVIDIA#4310) (NVIDIA#4373)
9a33422 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker pushed a commit to jon-barker/Megatron-LM that referenced this pull request Jul 10, 2026
@chtruong814 @kunlunl
Kunlun Li Jon Barker
cp: Fix UT timeout (NVIDIA#4310) (NVIDIA#4373)
d1af1ba 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
@chtruong814 @kunlunl
Kunlun Li @terminator123
cp: Fix UT timeout (NVIDIA#4310) (NVIDIA#4373)
06b1b39 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ko3n1g ko3n1g ko3n1g approved these changes

Assignees

No one assigned

Labels

complexity: medium

Projects

None yet

Milestone

Core 0.16

Development

Successfully merging this pull request may close these issues.

4 participants

@chtruong814 @svcnvidia-nemo-ci @ko3n1g @kunlunl