Skip to content

Disable MSC by default; opt in via --enable-msc - #4629

Merged
ericharper merged 6 commits into
NVIDIA:mainfrom
asolergi-nv:msc_disable
May 16, 2026
Merged

Disable MSC by default; opt in via --enable-msc#4629
ericharper merged 6 commits into
NVIDIA:mainfrom
asolergi-nv:msc_disable

Conversation

@asolergi-nv

@asolergi-nv asolergi-nv commented May 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • MSC integration is now disabled by default, even when multistorageclient is installed.
  • Replaces the --disable-msc flag with --enable-msc (opt-in).
  • Adds a warning in _get_filesystem_reader when MSC is enabled and cache_metadata=True is requested, since the MSC reader silently bypasses CachedMetadataFileSystemReader (so ckpt_assume_constant_structure has no effect).
  • Updates MSC_Integration.md to document the new opt-in model and the cache-metadata caveat.

Motivation

When MSC was enabled (the previous default), the dist-checkpoint loader silently swapped in msc.torch.MultiStorageFileSystemReader instead of CachedMetadataFileSystemReader, dropping the metadata cache without telling the user. Making MSC opt-in avoids this surprise on local-filesystem workloads and surfaces a warning when the trade-off does apply.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

@asolergi-nv
asolergi-nv requested review from a team as code owners May 5, 2026 15:15
@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft May 5, 2026 15:15
@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@copy-pr-bot

copy-pr-bot Bot commented May 5, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@asolergi-nv

asolergi-nv commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

claude[bot]
claude Bot reviewed May 5, 2026
View reviewed changes
Comment thread megatron/training/arguments.py
claude[bot]
claude Bot approved these changes May 5, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the flag flip and warning logic are correct, no stale references remain, and docs are accurate. One minor comment about adding test coverage for the two new code paths (RuntimeError on missing package, warning on cache_metadata bypass).

@asolergi-nv

asolergi-nv commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7263eca

@asolergi-nv
asolergi-nv marked this pull request as ready for review May 5, 2026 15:46
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team May 5, 2026 15:46
dimapihtar
dimapihtar reviewed May 14, 2026
View reviewed changes
Comment thread megatron/training/arguments.py
@dimapihtar
dimapihtar self-requested a review May 14, 2026 15:50
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Final Review PR is in the "final review" stage label May 14, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added Approved All necessary approvals have been made and removed Final Review PR is in the "final review" stage labels May 14, 2026
@ericharper
ericharper enabled auto-merge May 14, 2026 21:26
@asolergi-nv

asolergi-nv commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f0a286d

@asolergi-nv

asolergi-nv commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test c3b58fa

@ericharper
ericharper added this pull request to the merge queue May 15, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 15, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25923641822

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 15, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25926175989

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 15, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25928460491

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 15, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25938124252

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 15, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25942161275

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 16, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25947849841

Merged via the queue into NVIDIA:main with commit 95bba94 May 16, 2026
66 of 70 checks passed
Victarry pushed a commit to yanring/Megatron-LM that referenced this pull request May 18, 2026
Dennis Liu
Merge remote-tracking branch 'origin/main' into pr3431-conflict-fix
fc1ec88 
* origin/main: (138 commits)
  Refactor CUDA graph API: decompose cuda_graph_scope into full_iteration impl, inference scope, and per-layer capture modules (NVIDIA#4292)
  Add high-priority A2A stream and HybridEP preprocessing SMs (NVIDIA#4694)
  add is_torch_min_version in fsdp src (NVIDIA#4812)
  [Main][feat] Support A2A Overlap for Megatron-FSDP (NVIDIA#3797)
  Reorder mtp_post_process after attention backward in 1F1B schedule plan (NVIDIA#4695)
  [fix] Use MSC for checking checkpoint existence (NVIDIA#4251)
  Combine GEMM + SwiGLU fused MLP PRs (3890, 4071, 4095, 4219, 4311, 4324) → main (NVIDIA#4636)
  Strengthen test_checkpoint to verify distributed checkpoint behavior (NVIDIA#4711)
  Disable MSC by default; opt in via --enable-msc (NVIDIA#4629)
  additional tests for nvrx (NVIDIA#4522)
  Update copy-pr-bot.yaml [skip ci]
  ci: tolerate git-gc race in /home/runner chown after checkout (NVIDIA#4808)
  Inference: Optimize Prefill Engine Steps for Nemotron (NVIDIA#4764)
  chore: Update nightly tests golden values (NVIDIA#4805)
  ci: Update workflow to use same commit for building docker image and running tests (NVIDIA#4787)
  Update owners (NVIDIA#4794)
  Bump nvidia-modelopt>=0.44.0 (NVIDIA#4803)
  fix tokenizers in respect to newer transformers (NVIDIA#4608)
  Use Protocols to type-check linear_proj submodules of Attention (NVIDIA#3434)
  Fix recompute checkpointing + training CGs (NVIDIA#3919)
  ...

# Conflicts:
#	megatron/core/transformer/moe/moe_utils.py
@svcnvidia-nemo-ci svcnvidia-nemo-ci mentioned this pull request May 20, 2026
Closed
janEbert pushed a commit to janEbert/Megatron-LM that referenced this pull request Jun 2, 2026
@asolergi-nv @janEbert
Disable MSC by default; opt in via --enable-msc (NVIDIA#4629)
faa8307
yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026
@asolergi-nv @yhgalaxy
Disable MSC by default; opt in via --enable-msc (NVIDIA#4629)
7b450fa 
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker pushed a commit to jon-barker/Megatron-LM that referenced this pull request Jul 10, 2026
@asolergi-nv Jon Barker
Disable MSC by default; opt in via --enable-msc (NVIDIA#4629)
0aaa9ad 
Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
@asolergi-nv @terminator123
Disable MSC by default; opt in via --enable-msc (NVIDIA#4629)
6cf91fb
svcnvidia-nemo-ci pushed a commit to dimapihtar/Megatron-LM that referenced this pull request Aug 4, 2026
@asolergi-nv @dimapihtar
Disable MSC by default; opt in via --enable-msc (NVIDIA#4629)
9ea9a24 
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jaredcasper jaredcasper jaredcasper approved these changes
@ericharper ericharper ericharper approved these changes
@dimapihtar dimapihtar dimapihtar approved these changes
@claude claude[bot] claude[bot] approved these changes

Assignees

No one assigned

Labels

Approved All necessary approvals have been made complexity: low

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@asolergi-nv @svcnvidia-nemo-ci @jaredcasper @ericharper @dimapihtar