fix: NVRx async compatibility and defer resiliency import

[sbak5]

What does this PR do ?

This PR tightens Megatron’s NVRx async checkpoint integration and removes
unnecessary resiliency dependencies from non-NVRx paths.

Summary:

• require the NVRx async checkpointing symbols Megatron actually depends on
• remove the local MCore fallback for CachedMetadataFileSystemReader
• route async save helpers through async-strategy-specific modules
• defer in-process restart imports until the feature is enabled
• avoid eager type-only NVRx imports at module import time
• fail fast on async signature mismatches instead of silently filtering kwargs

Validated with:

• PyPI nvidia-resiliency-ext showing expected incompatibility on old async APIs
• compatible NVRx commits including 15a851565a4ce846c04431ecb0cf09903ab4837e and
  1c7f1a8ea60599a28835b0caf6353550a38d2db8
• sync save with and without nvidia-resiliency-ext installed
• normal training imports when in-process restart is not enabled

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

Pre-checks

• I have added relevant unit tests
• I have added relevant functional tests
• I have added proper typing to my code Typing guidelines
• I have added relevant documentation
• I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

1. When your PR is ready, click Ready for Review.
2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
   • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

1. Add the oncall reviewer (optional reviewer)
2. Add required review teams based on your changes

See the contribution guide for more details.

[sbak5]

/ok to test 9891c42

[sbak5]

/ok to test e4de738

[dimapihtar]

/ok to test eb2b9ea

[dimapihtar]

LGTM. Thank you!

[asolergi-nv]

/claude review

[claude]

Bug: Every other default in this PR was changed from "nvrx" to "mcore" (fully_parallel.py, torch.py:_get_filesystem_reader, torch.py:load, serialization.py), but this one was missed. All current callers pass the argument explicitly so it's not broken today, but a future caller relying on the default would silently get the nvrx path — which may fail if nvidia-resiliency-ext isn't installed.

Suggested change

	writer, save_state_dict_ret, async_strategy: str = "nvrx"
	writer, save_state_dict_ret, async_strategy: str = "mcore"

[claude]

Light review — one likely bug flagged inline (default value inconsistency on get_save_and_finalize_callbacks).

Test coverage: The new nvrx.py module adds has_nvrx_async_support() (with 8 symbol checks across 4 submodules) and make_nvrx_async_request(), but neither has direct unit tests. The only new test covers a single get_async_strategy error path. Consider adding tests for:

• has_nvrx_async_support() returning False when individual symbols are missing (not just entire modules)
• make_nvrx_async_request() constructing requests correctly for both mcore and nvrx AsyncRequest classes
• The new init_persistent_async_worker signature-checking logic in async_utils.py

[asolergi-nv]

Fix default value!

[sbak5]

/ok to test 805473a

[svcnvidia-nemo-ci]

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24847833017

[svcnvidia-nemo-ci]

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24850965249

[svcnvidia-nemo-ci]

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24854872727