chore: nightly sync main into dev (08_04_2026)

.claude/skills/respond-to-issue/SKILL.md

@@ -0,0 +1,64 @@
+---
+name: respond-to-issue
+description: Research and draft a response to a GitHub issue or question from an external contributor. Use when the user shares a GitHub issue URL or asks to respond to a community question.
+user_invocable: true
+argument: "<github-issue-url-or-number>"
+---
+
+# Respond to GitHub Issue
+
+Help a maintainer draft a high-quality response to a GitHub issue from an external contributor.
+
+## Workflow
+
+### 1. Understand the issue
+
+- Fetch the issue using `gh issue view <number> --repo NVIDIA/Megatron-LM --json title,body,comments,labels,state`.
+- Read the title, body, and all existing comments to understand the full context.
+- Identify the type: bug report, feature request, question, or discussion.
+
+### 2. Research the codebase
+
+- Based on what the issue is asking, search the Megatron-LM codebase for the relevant code.
+- Read the relevant source files to understand the current behavior.
+- If the issue references specific files or functions, read those directly.
+- Check `git log --oneline -20 -- <relevant-files>` to see if there have been recent changes that address or relate to the issue.
+- Use `git log -S "<symbol>" --oneline` to trace when code was added or removed — this is especially useful for questions about unused/deprecated code or missing features.
+- If the issue is about a bug, try to confirm whether the reported behavior matches the code.
+- Check whether an existing PR already addresses the issue: `gh pr list --repo NVIDIA/Megatron-LM --search "<keywords>" --limit 5`.
+
+### 3. Verify before citing
+
+Before including specific details in the response, verify them:
+- If citing a commit hash, confirm the commit message and diff match what you're claiming (`git show <hash> --stat`).
+- If citing a file path and line number, re-read the file to confirm the line content is correct.
+- If claiming code is unused or missing, do a thorough grep to make sure you haven't missed a reference.
+
+### 4. Draft the response
+
+Write a response that:
+- Is technically accurate and grounded in the actual code (cite file paths and line numbers where helpful).
+- Is respectful and welcoming to external contributors.
+- Directly addresses the question or concern raised.
+- If the contributor identified a real gap or bug, acknowledge it clearly.
+- If there's a workaround, mention it.
+- If work is planned or a fix would be welcome, say so and suggest next steps (e.g., "a PR to address this would be welcome").
+- Keeps the tone professional but friendly.
+- Is concise -- contributors appreciate direct answers, not walls of text.
+
+### 5. Suggest follow-up actions
+
+If the issue identifies something cleanly actionable (dead code to remove, a small bug fix, a missing feature), tell the maintainer and offer to create a branch and PR to address it — don't just draft a comment.
+
+### 6. Present to the maintainer
+
+Show the drafted response to the user (the maintainer) for review. Do NOT post it to GitHub automatically. The maintainer will decide whether to post it, edit it, or ask for changes.
+
+Format the draft as a quoted markdown block so it's easy to copy.
+
+## Important guidelines
+
+- Never post comments to GitHub without explicit approval from the user.
+- If you're unsure about the answer, say so clearly in your draft and flag the uncertainty for the maintainer.
+- If the issue is outside the scope of what you can determine from the code, tell the maintainer what you found and what remains unclear.
+- Check whether similar issues exist that might be relevant: `gh issue list --repo NVIDIA/Megatron-LM --search "<keywords>" --limit 5`.

.cursorrules

@@ -0,0 +1 @@
+See CLAUDE.md for all repository guidelines.

.github/actions/action.yml

@@ -75,16 +75,12 @@ runs:
       shell: bash
       run: sudo chown -R $(whoami) /home/runner/
 
-    - name: Setup python
-      uses: actions/setup-python@v5
-      with:
-        python-version: 3.12
-
-    - name: Install uuidgen
+    - name: Setup python and install dependencies
       shell: bash -x -e -u -o pipefail {0}
       run: |
         apt-get update
-        apt-get install -y uuid-runtime
+        apt-get install -y python3.12 python3.12-venv uuid-runtime
+        curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/local/bin sh
 
     - name: Create run-script (unit test)
       shell: bash -x -e -u -o pipefail {0}
@@ -97,7 +93,6 @@ runs:
         export PYTHONPATH=$(pwd)
         export NEMORUN_HOME=$(pwd)
         export NCCL_DEBUG=INFO
-        pip install --no-cache-dir "uv<0.9.29"
         uv venv .venv
         uv cache clean
         uv sync --no-cache --only-group test
@@ -137,7 +132,6 @@ runs:
 
         export PYTHONPATH=$(pwd)
         export NEMORUN_HOME=$(pwd)
-        pip install --no-cache-dir "uv<0.9.29"
         uv venv .venv
         uv cache clean
         uv sync --no-cache --only-group test

.github/copy-pr-bot.yaml

@@ -1,4 +1,4 @@
 enabled: true
 auto_sync_draft: false
 auto_sync_ready: true
-trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "CarlosGomes98", "ChenhanYu", "FDecaYed", "HaochenYuan", "ISEEKYAN", "JRD971000", "Phlip79", "QiZhangNV", "RPrenger", "ShriyaRishab", "Victarry", "Wohox", "ZhiyuLi-Nvidia", "ahmadki", "aklife97", "ananthsub", "asolergi-nv", "buptzyb", "chtruong814", "cjld", "cspades", "cuichenx", "deepakn94", "dimapihtar", "dingqingy-nv", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "faradawn", "frsun-nvda", "gautham-kollu", "gdengk", "guyueh1", "huvunvidia", "hxbai", "ilml", "jalbericiola", "janEbert", "jaredcasper", "jenchen13", "jiemingz", "jingqiny-99", "jkamalu", "jon-barker", "jstjohn", "kajalj22", "kanz-nv", "kevalmorabia97", "ko3n1g", "ksivaman", "kunlunl", "kvareddy", "kwyss-nvidia", "layalir", "lhb8125", "lmcafee-nvidia", "maanug-nv", "mathemakitten", "matthieule", "mchrzanowski", "mehraakash", "mkhona-nvidia", "nanz-nv", "parthmannan", "prajwal1210", "pthombre", "rhewett-nv", "rogerwaleffe", "sajadn", "sanandaraj5597", "sancha", "santhnm2", "sbak5", "shanmugamr1992", "sharathts", "shengf-nv", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sudhakarsingh27", "tdene", "theothermike", "thomasdhc", "tomlifu", "trintamaki", "tylerpoon", "wdykas", "wplf", "xiaoyao0115", "xuwchen", "yanring", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yueshen2016", "yuzhongw-nvidia", "zhongbozhu"]
+trustees_override: ["AAnoosheh", "ArEsKay3", "Autumn1998", "BestJuly", "BoxiangW", "CarlosGomes98", "ChenhanYu", "FDecaYed", "HaochenYuan", "ISEEKYAN", "JRD971000", "Phlip79", "QiZhangNV", "RPrenger", "ShriyaRishab", "Victarry", "WanZzzzzz", "Wohox", "ZhiyuLi-Nvidia", "ahmadki", "aklife97", "ananthsub", "asolergi-nv", "buptzyb", "chtruong814", "cjld", "cspades", "cuichenx", "deepakn94", "dimapihtar", "dingqingy-nv", "duncanriach", "erhoo82", "ericharper", "fanshiqing", "faradawn", "frsun-nvda", "gautham-kollu", "gdengk", "guyueh1", "hexinw-nvidia", "huvunvidia", "hxbai", "ilml", "jalbericiola", "janEbert", "jaredcasper", "jenchen13", "jiemingz", "jingqiny-99", "jkamalu", "jon-barker", "jstjohn", "kajalj22", "kanz-nv", "keshavb96", "kevalmorabia97", "ko3n1g", "ksivaman", "kunlunl", "kvareddy", "kwyss-nvidia", "layalir", "lhb8125", "lmcafee-nvidia", "maanug-nv", "mathemakitten", "matthieule", "mchrzanowski", "mehraakash", "minitu", "mkhona-nvidia", "nanz-nv", "parthmannan", "prajwal1210", "pthombre", "rhewett-nv", "rogerwaleffe", "sajadn", "sanandaraj5597", "sancha", "santhnm2", "sbak5", "shanmugamr1992", "sharathts", "sheliang-nv", "shengf-nv", "shifangx", "shjwudp", "sidsingh-nvidia", "skyw", "sudhakarsingh27", "tdene", "theothermike", "thomasdhc", "tomlifu", "trintamaki", "tylerpoon", "wdykas", "wplf", "xiaoyao0115", "xuwchen", "yanring", "yaox12", "yaoyu-33", "yashaswikarnati", "yeyu-nvidia", "yobibyte", "youngeunkwon0405", "yueshen2016", "yuzhongw-nvidia", "zhongbozhu"]

.github/oncall_schedule.json

@@ -1,12 +1,4 @@
 [
-    {
-        "user": "janEbert",
-        "date": "2026-03-25"
-    },
-    {
-        "user": "gautham-kollu",
-        "date": "2026-04-01"
-    },
     {
         "user": "ilml",
         "date": "2026-04-08"
@@ -20,31 +12,47 @@
         "date": "2026-04-22"
     },
     {
-        "user": "BoxiangW",
+        "user": "maanug-nv",
         "date": "2026-04-29"
     },
     {
-        "user": "maanug-nv",
+        "user": "dimapihtar",
         "date": "2026-05-06"
     },
     {
-        "user": "dimapihtar",
+        "user": "gautham-kollu",
         "date": "2026-05-13"
     },
     {
-        "user": "gautham-kollu",
+        "user": "janEbert",
         "date": "2026-05-20"
     },
     {
         "user": "ilml",
         "date": "2026-05-27"
     },
     {
-        "user": "janEbert",
+        "user": "Phlip79",
         "date": "2026-06-03"
     },
     {
-        "user": "maanug-nv",
+        "user": "asolergi-nv",
         "date": "2026-06-10"
+    },
+    {
+        "user": "maanug-nv",
+        "date": "2026-06-17"
+    },
+    {
+        "user": "dimapihtar",
+        "date": "2026-06-24"
+    },
+    {
+        "user": "gautham-kollu",
+        "date": "2026-07-01"
+    },
+    {
+        "user": "janEbert",
+        "date": "2026-07-08"
     }
 ]

.github/workflows/_build_test_publish_wheel.yml

@@ -71,10 +71,10 @@ jobs:
           pushd $BUILD_DIR
             rm LICENSE || true
             docker run --rm -v $(pwd):/workspace -w /workspace $IMAGE bash -c '\
-              for python_version in cp310 cp311 cp312 cp313; do \
+              for python_version in cp311 cp312 cp313; do \
                 /opt/python/${python_version}-${python_version}/bin/pip install --upgrade "setuptools<80.0.0,>=77.0.0" build; \
               done && \
-              for python_version in cp310 cp311 cp312 cp313; do \
+              for python_version in cp311 cp312 cp313; do \
                 /opt/python/${python_version}-${python_version}/bin/python -m build; \
               done \
             '
@@ -118,22 +118,20 @@ jobs:
 
           if [ "$PACKAGE" = "megatron-core" ]; then
             if [[ "$PLATFORM" == "arm64" ]]; then
-              for file in dist/$WHEEL_PREFIX*cp310*aarch64.whl; do
-                pip install --no-cache-dir "$file"
-              done
+              WHEEL_GLOB="dist/${WHEEL_PREFIX}*cp312*aarch64.whl"
             else
-              for file in dist/$WHEEL_PREFIX*cp310*x86_64.whl; do
-                pip install --no-cache-dir "$file"
-              done
+              WHEEL_GLOB="dist/${WHEEL_PREFIX}*cp312*x86_64.whl"
             fi
           else
-            pip install --no-cache-dir dist/$WHEEL_PREFIX*.whl
+            WHEEL_GLOB="dist/${WHEEL_PREFIX}*.whl"
           fi
 
-          sudo rm -rf megatron/
-
-          RELEASE_NUMBER=$(python -c "import $ROOTPATH; print($ROOTPATH.__version__)")
-          test "${{ steps.build-wheel.outputs.expected-release-number }}" == "$RELEASE_NUMBER"
+          docker run --rm -v $(pwd):/workspace -w /workspace $IMAGE bash -c "\
+            /opt/python/cp312-cp312/bin/pip install --no-cache-dir $WHEEL_GLOB && \
+            rm -rf megatron/ && \
+            RELEASE_NUMBER=\$(/opt/python/cp312-cp312/bin/python -c 'import $ROOTPATH; print($ROOTPATH.__version__)') && \
+            test '${{ steps.build-wheel.outputs.expected-release-number }}' == \"\$RELEASE_NUMBER\" \
+          "
 
       - name: Upload wheels
         uses: actions/upload-artifact@v6

.github/workflows/claude_review.yml

@@ -33,6 +33,12 @@ jobs:
           fetch-depth: 1
           ref: ${{ steps.get-pr-head-commit.outputs.sha }}
 
+      - name: React to trigger comment
+        run: |
+          gh api repos/$REPO/issues/comments/${{ github.event.comment.id }}/reactions \
+            --method POST \
+            -f content='eyes'
+
       - name: Run Claude Code Review
         uses: anthropics/claude-code-action@v1
         with:
@@ -52,6 +58,8 @@ jobs:
             - Critical bugs or logic errors
             - Typos in code, comments, or strings
             - Missing or insufficient test coverage for changed code
+              - If the PR adds a new feature or significant functionality without corresponding tests, suggest adding tests
+              - If the PR fixes a bug that was not caught by an existing unit test, suggest adding a regression test to prevent recurrence
             - Outdated or inaccurate documentation affected by the changes
 
             Do NOT comment on:

.github/workflows/release-freeze.yml

@@ -34,13 +34,16 @@ on:
         default: true
 jobs:
   code-freeze:
-    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_code_freeze.yml@v0.22.5
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_code_freeze.yml@v0.86.0
     with:
-      library-name: Megatron-Bridge
-      python-package: megatron.bridge
+      library-name: Megatron-Core
+      python-package: megatron.core
       release-type: ${{ inputs.release-type }}
       freeze-commit: ${{ inputs.freeze-commit }}
       dry-run: ${{ inputs.dry-run }}
+      release-branch-prefix: core_
+      use-pat: true
     secrets:
-      SLACK_WEBHOOK: ${{ secrets.SLACK_MAIN_CHANNEL_WEBHOOK }}
+      SLACK_WEBHOOK: ${{ inputs.dry-run && secrets.SLACK_CI_CHANNEL_WEBHOOK ||secrets.SLACK_MAIN_CHANNEL_WEBHOOK }}
       SLACK_WEBHOOK_ADMIN: ${{ secrets.SLACK_TEAM_GROUP_ID }}
+      PAT: ${{ secrets.PAT }}

AGENTS.md

@@ -0,0 +1 @@
+See [CLAUDE.md](CLAUDE.md) for all repository guidelines.

CLAUDE.md

@@ -0,0 +1,13 @@
+# Repository Guidelines
+
+## Contributing
+
+### Pull Requests
+
+- All PRs must be created as **drafts**. Use `gh pr create --draft` or the GitHub UI draft option.
+- Never push branches directly to `https://github.com/NVIDIA/Megatron-LM`. You must push your branch to a personal fork (e.g. `https://github.com/<your-username>/Megatron-LM`), then open a PR from the fork's branch against `NVIDIA/Megatron-LM`.
+- Read [docs/developer/contribute.md](docs/developer/contribute.md) for the full contribution policy, including code style, commit message conventions, and issue guidelines.
+
+### Code Quality
+
+- After editing imports in any Python files, always run `uv run isort` on those files to fix import order before committing.

docker/Dockerfile.ci.dev

@@ -2,7 +2,7 @@
 # syntax=docker/dockerfile:1.3-labs
 
 ARG FROM_IMAGE_NAME
 FROM ${FROM_IMAGE_NAME} as main
 ENV PIP_CONSTRAINT=""
 ENV DEBIAN_FRONTEND=noninteractive
 ARG UV_VERSION=0.7.2
@@ -16,7 +16,7 @@
 
 RUN bash -ex <<"EOF"
     apt-get update
-    apt-get install -y --no-install-recommends gettext python3-venv psmisc uuid-runtime 
+    apt-get install -y --no-install-recommends gettext python3-venv psmisc uuid-runtime
     apt-get clean
     python -m venv /opt/jet
     ARCH=$(uname -m)
@@ -31,21 +31,17 @@
     curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
 EOF
 
-RUN ln -sf /usr/local/cuda/targets/x86_64-linux/include/cuda \
-    /usr/local/include/cuda
-RUN find /usr/local/cuda -name "utility" 2>/dev/null | head -5 && \
-    ls /usr/local/cuda/targets/x86_64-linux/include/ | head -20
-
 COPY README.md pyproject.toml uv.lock /workspace/
 COPY megatron/core/__init__.py /workspace/megatron/core/
 COPY megatron/core/package_info.py /workspace/megatron/core/
 ARG IMAGE_TYPE=dev
+ENV NVTE_BUILD_NUM_PHILOX_ROUNDS=3
 RUN --mount=type=cache,target=/root/.cache/uv \
     bash -ex <<"EOF"
     export NVTE_CUDA_ARCHS="80;90;100"
     uv venv ${UV_PROJECT_ENVIRONMENT} --system-site-packages
     uv sync --only-group build
-    uv sync --extra ${IMAGE_TYPE} --extra mlm --group no_pypi_wheels --link-mode copy --locked \
+    uv sync --extra ${IMAGE_TYPE} --extra mlm --link-mode copy --locked \
         --no-install-package torch \
         --no-install-package torchvision \
         --no-install-package triton \
@@ -86,7 +82,7 @@
 ENV UV_PYTHON=$UV_PROJECT_ENVIRONMENT/bin/python
 
 ##### For NVIDIANS only #####
 FROM main as jet
 ARG JET_API_VERSION
 ENV PATH="$PATH:/opt/jet/bin"
 RUN --mount=type=secret,id=JET_INDEX_URLS bash -ex <<"EOF"

docs/api-guide/core/dist_checkpointing.md

@@ -123,6 +123,42 @@ You can combine formats to optimize both flexibility and performance:
    3. Save at least one checkpoint under the new model parallel configuration.
    4. (Optional) To continue the training with updated model parallelism and better checkpointing performance, stop training and switch back to ``dp_reshardable`` format by removing ``--dist-ckpt-optim-fully-reshardable``.
 
+## Async Checkpoint Saving Strategy
+
+The framework supports asynchronous checkpoint saving to improve training performance by offloading I/O operations. 
+
+We are transitioning to a new async saving implementation based on the **NVRx (NVIDIA Resiliency Extension)** package. The legacy async strategy (referred to as **mcore**) is being deprecated.
+
+### Migration to NVRx
+
+- The **NVRx-based async strategy** will become the **default** in mcore v0.17.
+- The existing **mcore async strategy** is **deprecated** and will be removed in future versions.
+- A deprecation warning is emitted when using the mcore strategy.
+
+### Selecting Async Strategy
+
+`--async-strategy` flag is introduced to control the async strategy. To use legacy async strategy (**mcore**), set:
+
+```bash
+--async-strategy mcore
+```
+
+### NVRx Dependency
+
+To use the NVRx-based async strategy, you must install the `nvidia-resiliency-ext` package.
+
+```bash
+git clone https://github.com/NVIDIA/nvidia-resiliency-ext
+cd nvidia-resiliency-ext
+pip install .
+```
+
+> NOTE
+
+- If `nvidia-resiliency-ext` is not installed, the NVRx async strategy will not be available.
+- The `mcore` strategy will remain temporarily to ensure a smooth transition but will be removed in future releases.
+- It is strongly recommended to migrate to the NVRx strategy as soon as possible.
+
 ## Subpackages
 
 ```{toctree}

docs/get-started/install.md

@@ -58,7 +58,7 @@ uv pip install --no-build-isolation "megatron-core[training,dev]"
 ```
 
 ```{warning}
-Building from source can consume a large amount of memory. By default the build runs one compiler job per CPU core, which may cause out-of-memory failures on machines with many cores. To limit parallel compilation jobs, set the `MAX_JOBS` environment variable before installing (e.g. `MAX_JOBS=4`).
+Building from source can consume a large amount of memory. By default the build runs one compiler job per CPU core, which can cause out-of-memory failures on machines with many cores. To limit parallel compilation jobs, set the `MAX_JOBS` environment variable before installing (for example, `MAX_JOBS=4`).
 ```
 
 ```{tip}
@@ -109,7 +109,7 @@ docker run --gpus all -it --rm \
 ```
 
 ```{note}
-The NGC PyTorch container constrains the Python environment globally via `PIP_CONSTRAINT`. The `-e PIP_CONSTRAINT=` flag above unsets this so that Megatron Core and its dependencies install correctly.
+The NGC PyTorch container constrains the Python environment globally using `PIP_CONSTRAINT`. The `-e PIP_CONSTRAINT=` flag above unsets this so that Megatron Core and its dependencies install correctly.
 ```
 
 Then install Megatron Core inside the container (torch is already available in the NGC image):
@@ -120,4 +120,4 @@ uv pip install --no-build-isolation "megatron-core[training,dev]"
 ```
 
 
-You are now ready to run training. See [Your First Training Run](quickstart.md) for next steps.
+You are now ready to run training. Refer to [Your First Training Run](quickstart.md) for next steps.