Reduce-scatter implementation with FP32 accumulation - #1967
Merged
deepakn94 merged 1 commit intoOct 30, 2025
Conversation
deepakn94
commented
Oct 27, 2025
deepakn94
commented
Contributor
- Include plumbing to use this implementation for actual model training
- Add async_op=True option for reduce_scatter_with_fp32_accumulation
- Add unit test
Contributor
Author
|
@ko3n1g , do I need to do anything here to run unit tests? |
kvareddy
approved these changes
Oct 28, 2025
Contributor
Author
|
/ok to test eaba87c |
deepakn94
force-pushed
the
dnarayanan/rs_bf16comm_fp32accumulate
branch
2 times, most recently
from
October 30, 2025 05:07
0db4223 to
be7b3b8
Compare
- Include plumbing to use this implementation for actual model training - Add async_op=True option for reduce_scatter_with_fp32_accumulation - Add unit test Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
deepakn94
force-pushed
the
dnarayanan/rs_bf16comm_fp32accumulate
branch
from
October 30, 2025 06:01
be7b3b8 to
199b11b
Compare
ko3n1g
approved these changes
Oct 30, 2025
asolergi-nv
pushed a commit
to asolergi-nv/Megatron-LM
that referenced
this pull request
Oct 31, 2025
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
sidsingh-nvidia
pushed a commit
to sidsingh-nvidia/Megatron-LM
that referenced
this pull request
Oct 31, 2025
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
yeyu-nvidia
pushed a commit
to yeyu-nvidia/Megatron-LM
that referenced
this pull request
Oct 31, 2025
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Jianbing-D
pushed a commit
to Jianbing-D/Megatron-LM
that referenced
this pull request
Nov 12, 2025
* ci: Move test optimizer into its own bucket (NVIDIA#1909) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Use matrix for approval-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update function name Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Adjust approval-bot for copy-pr-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Remove attribute Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update container image tag to use GitHub SHA * chore: Remove file * ci: Fix approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Configure cherrypick bot (NVIDIA#1925) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ci approve dev (NVIDIA#1933) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update nightly schedule (NVIDIA#1934) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Bump pre-flight for runs on main/dev (NVIDIA#1935) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Allow skipping on main (NVIDIA#1936) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/pr template community bot (NVIDIA#1937) * ci: More granular unit tests buckets (NVIDIA#1932) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add sequence packing to RL (NVIDIA#1911) Add sequence packing to RL * chore: Update template (NVIDIA#1939) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Add description about who can merge (NVIDIA#1940) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/fix main on eos (NVIDIA#1938) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/internal mrs (NVIDIA#1942) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix branch of approval bot (NVIDIA#1944) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approvalbot for other branches (NVIDIA#1947) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval bot (NVIDIA#1949) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate rule Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update golden values nightly Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Smaller image Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Better output Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix sync bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/sync branches (NVIDIA#1956) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Increase time limit for main tests Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/add milestone (NVIDIA#1951) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove M-FSDP testing under LTS environment (NVIDIA#1959) * ci: Run on push to release branch (NVIDIA#1960) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Add golden values for inference Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix typo in rl section of CODEOWNERS (NVIDIA#1968) * ci: Update copyright checker (NVIDIA#1973) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/auto reminder GitHub (NVIDIA#1955) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update secret Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): `Run tests` label (NVIDIA#1970) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable tests again Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Add merge-group to copyright check Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Copyright check on merge-queue Signed-off-by: oliver könig <okoenig@nvidia.com> * zarr soft deprecation (NVIDIA#2004) Signed-off-by: dimapihtar <dpihtar@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Make `get_asyncio_loop` safe to use repeatedly (NVIDIA#1990) Co-authored-by: oliver könig <okoenig@nvidia.com> * Update symmetric registration interface to sync-up with upstream pytorch change (NVIDIA#1924) Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Update codeowners (NVIDIA#2012) Signed-off-by: oliver könig <okoenig@nvidia.com> * Deduplicate dynamic engine + coordinator. (NVIDIA#1981) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Safely access state dict args in load ckpt (NVIDIA#1957) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Allow mixed-batch sampling in dynamic inference (NVIDIA#1927) * Stop Nemo_CICD_Test from failing in forks (NVIDIA#2024) * Clean up dynamic inference step (NVIDIA#1992) Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Auto-update copy-pr-bot vetters (NVIDIA#1850) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> * Have datasets account for tokenizers which incorrectly define PAD (NVIDIA#2017) * ci: Enable integration tests (NVIDIA#2023) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix build-push-wheel workflow (NVIDIA#2022) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update tooling for interactive jobs (NVIDIA#2032) Signed-off-by: oliver könig <okoenig@nvidia.com> * revert(hotfix): ci: trustees_override (NVIDIA#2041) Signed-off-by: oliver könig <okoenig@nvidia.com> * add missing warnings import in model parallel config (NVIDIA#2039) Signed-off-by: ykarnati <ykarnati@nvidia.com> * Reduce-scatter implementation with FP32 accumulation (NVIDIA#1967) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> * ci(fix): Workflows on `main` (NVIDIA#2045) Signed-off-by: oliver könig <okoenig@nvidia.com> * build: Bump modelopt (NVIDIA#2046) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove TestCaptureFreezeGC unit test. (NVIDIA#1978) * ci: Add multi-approval action (NVIDIA#2051) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Repair codeowners file * ci(hotfix): Set docs allowed to fail Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/test iteration time (NVIDIA#2067) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove performance for ckpt-resume Signed-off-by: oliver könig <okoenig@nvidia.com> * Allow inference test throughput to vary by 10% (NVIDIA#2070) * ci(hotfix): Inference test pipeline Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Fix autoformatter (NVIDIA#2073) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove iteration-time from t5 Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Bypass approvalbot in merge-queue (NVIDIA#2082) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Enable merge-group for approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update local tooling (NVIDIA#2066) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add extra RL files (NVIDIA#2077) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Prevent summary jobs from running in forks (NVIDIA#2083) Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Fix test scope (NVIDIA#2091) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove publish workflows Signed-off-by: oliver könig <okoenig@nvidia.com> * Refactor the attention metadata into separate classes (NVIDIA#2001) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Guard against incorrectly using MoE prefill graphs (NVIDIA#2030) Co-authored-by: oliver könig <okoenig@nvidia.com> * Revert "Refactor the attention metadata into separate classes (NVIDIA#2001)" This reverts commit a652e2c. * Run mr-slim tests in lightweight-mode (NVIDIA#2106) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Inference | Lazy compile UVM allocator. (NVIDIA#1977) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Reenable trustees (NVIDIA#2108) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Inference | Lazy compile UVM allocator. (NVIDIA#1977)" This reverts commit 7487c53. * ci(fix): Changeset of copyright checker (NVIDIA#2110) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/chore/update release settings (NVIDIA#2097) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove unnecessary check on rotary_pos_cos (NVIDIA#2003) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * (Reverted) Inference | Lazy compile UVM allocator. (NVIDIA#2125) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor Attention Metadata to Separate Classes (NVIDIA#2112) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor model_provider to model_builder format for ModelOpt examples (NVIDIA#2107) * wandb Inference stats logging (NVIDIA#2026) Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> * Make `PipelineParallelLayout` always return str from ` __repr__` (NVIDIA#2055) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Add flash_attn_3 as first option for FA3 import (NVIDIA#2010) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Add debugging hint for case when cudagraphs are created but no matching runner is found (NVIDIA#2129) * ci: LTS container (NVIDIA#2133) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "ci: LTS container (NVIDIA#2133)" This reverts commit eb48e81. * Fix param init (NVIDIA#2033) Signed-off-by: Chen Cui <chcui@nvidia.com> * Hotfix to unit tests on hopper FA3 (NVIDIA#2143) * Add BytesIO to safe_globals (NVIDIA#2074) * add deprecation warning for legacy tokenizer system (NVIDIA#2145) Signed-off-by: dimapihtar <dpihtar@gmail.com> * replay: ci: Bump LTS container (NVIDIA#2157) Signed-off-by: oliver könig <okoenig@nvidia.com> * Hotfix to unit tests on hopper FA3 (bis) (NVIDIA#2179) * Fix has_modelopt_state() for native Torch checkpoint format (NVIDIA#2160) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * chore: Remove codeowners (NVIDIA#2175) Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix FP8 inference with sequence parallelism (NVIDIA#2009) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Replace ModelOpt generation server (NVIDIA#2147) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * Add hybrid model support for dynamic inference engine (NVIDIA#1907) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Async task and event loop safety in Megatron Core (NVIDIA#2025) Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> * Rename skip_prompt_log_probs (NVIDIA#2181) * Dynamic inference context | UVM only. (NVIDIA#1983) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Update copy-pr-bot.yaml [skip ci] Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Dynamic inference context | UVM only. (NVIDIA#1983)" This reverts commit d6979d6. * ci: Run `auto-update-copy-pr-bot` only on forks (NVIDIA#2191) Signed-off-by: oliver könig <okoenig@nvidia.com> * Inference throughput tests: refactor goldens to be in list format (NVIDIA#2072) * Enable TE custom quantization recipe (NVIDIA#2005) Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: root <Evgeny> * Remove redundant logits calculations in gpt_model --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com> Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com> Co-authored-by: helen ngo <helenn@nvidia.com> Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: kanz-nv <kanz@nvidia.com> Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com> Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> Co-authored-by: Evgeny Tsykunov <e.tsykunov@gmail.com>
Jianbing-D
pushed a commit
to Jianbing-D/Megatron-LM
that referenced
this pull request
Nov 12, 2025
* ci: Move test optimizer into its own bucket (NVIDIA#1909) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Use matrix for approval-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update function name Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Adjust approval-bot for copy-pr-bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Parametrize workflow Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Remove attribute Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update container image tag to use GitHub SHA * chore: Remove file * ci: Fix approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Configure cherrypick bot (NVIDIA#1925) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ci approve dev (NVIDIA#1933) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update nightly schedule (NVIDIA#1934) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Bump pre-flight for runs on main/dev (NVIDIA#1935) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Allow skipping on main (NVIDIA#1936) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/pr template community bot (NVIDIA#1937) * ci: More granular unit tests buckets (NVIDIA#1932) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add sequence packing to RL (NVIDIA#1911) Add sequence packing to RL * chore: Update template (NVIDIA#1939) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Add description about who can merge (NVIDIA#1940) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/fix main on eos (NVIDIA#1938) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/internal mrs (NVIDIA#1942) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix branch of approval bot (NVIDIA#1944) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approvalbot for other branches (NVIDIA#1947) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval bot (NVIDIA#1949) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate rule Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update golden values nightly Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval gate Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Smaller image Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Better output Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: sync branches Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix sync bot Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Finalize Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/sync branches (NVIDIA#1956) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Increase time limit for main tests Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/add milestone (NVIDIA#1951) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove M-FSDP testing under LTS environment (NVIDIA#1959) * ci: Run on push to release branch (NVIDIA#1960) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Add golden values for inference Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix typo in rl section of CODEOWNERS (NVIDIA#1968) * ci: Update copyright checker (NVIDIA#1973) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/auto reminder GitHub (NVIDIA#1955) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Update secret Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(fix): `Run tests` label (NVIDIA#1970) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable tests again Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Add merge-group to copyright check Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Copyright check on merge-queue Signed-off-by: oliver könig <okoenig@nvidia.com> * zarr soft deprecation (NVIDIA#2004) Signed-off-by: dimapihtar <dpihtar@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Make `get_asyncio_loop` safe to use repeatedly (NVIDIA#1990) Co-authored-by: oliver könig <okoenig@nvidia.com> * Update symmetric registration interface to sync-up with upstream pytorch change (NVIDIA#1924) Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Update codeowners (NVIDIA#2012) Signed-off-by: oliver könig <okoenig@nvidia.com> * Deduplicate dynamic engine + coordinator. (NVIDIA#1981) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Safely access state dict args in load ckpt (NVIDIA#1957) Signed-off-by: Maanu Grover <maanug@nvidia.com> * Allow mixed-batch sampling in dynamic inference (NVIDIA#1927) * Stop Nemo_CICD_Test from failing in forks (NVIDIA#2024) * Clean up dynamic inference step (NVIDIA#1992) Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Auto-update copy-pr-bot vetters (NVIDIA#1850) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> * Have datasets account for tokenizers which incorrectly define PAD (NVIDIA#2017) * ci: Enable integration tests (NVIDIA#2023) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci: Fix build-push-wheel workflow (NVIDIA#2022) Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update tooling for interactive jobs (NVIDIA#2032) Signed-off-by: oliver könig <okoenig@nvidia.com> * revert(hotfix): ci: trustees_override (NVIDIA#2041) Signed-off-by: oliver könig <okoenig@nvidia.com> * add missing warnings import in model parallel config (NVIDIA#2039) Signed-off-by: ykarnati <ykarnati@nvidia.com> * Reduce-scatter implementation with FP32 accumulation (NVIDIA#1967) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> * ci(fix): Workflows on `main` (NVIDIA#2045) Signed-off-by: oliver könig <okoenig@nvidia.com> * build: Bump modelopt (NVIDIA#2046) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove TestCaptureFreezeGC unit test. (NVIDIA#1978) * ci: Add multi-approval action (NVIDIA#2051) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Repair codeowners file * ci(hotfix): Set docs allowed to fail Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/ci/test iteration time (NVIDIA#2067) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove performance for ckpt-resume Signed-off-by: oliver könig <okoenig@nvidia.com> * Allow inference test throughput to vary by 10% (NVIDIA#2070) * ci(hotfix): Inference test pipeline Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Fix autoformatter (NVIDIA#2073) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove iteration-time from t5 Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Disable inference test Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Bypass approvalbot in merge-queue (NVIDIA#2082) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Enable merge-group for approval bot Signed-off-by: oliver könig <okoenig@nvidia.com> * chore: Update local tooling (NVIDIA#2066) Signed-off-by: oliver könig <okoenig@nvidia.com> * Add extra RL files (NVIDIA#2077) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Prevent summary jobs from running in forks (NVIDIA#2083) Co-authored-by: oliver könig <okoenig@nvidia.com> * ci: Fix test scope (NVIDIA#2091) Signed-off-by: oliver könig <okoenig@nvidia.com> * ci(hotfix): Remove publish workflows Signed-off-by: oliver könig <okoenig@nvidia.com> * Refactor the attention metadata into separate classes (NVIDIA#2001) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Guard against incorrectly using MoE prefill graphs (NVIDIA#2030) Co-authored-by: oliver könig <okoenig@nvidia.com> * Revert "Refactor the attention metadata into separate classes (NVIDIA#2001)" This reverts commit a652e2c. * Run mr-slim tests in lightweight-mode (NVIDIA#2106) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Inference | Lazy compile UVM allocator. (NVIDIA#1977) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * chore: Reenable trustees (NVIDIA#2108) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Inference | Lazy compile UVM allocator. (NVIDIA#1977)" This reverts commit 7487c53. * ci(fix): Changeset of copyright checker (NVIDIA#2110) Signed-off-by: oliver könig <okoenig@nvidia.com> * Ko3n1g/chore/update release settings (NVIDIA#2097) Signed-off-by: oliver könig <okoenig@nvidia.com> * Remove unnecessary check on rotary_pos_cos (NVIDIA#2003) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> * (Reverted) Inference | Lazy compile UVM allocator. (NVIDIA#2125) Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor Attention Metadata to Separate Classes (NVIDIA#2112) Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Refactor model_provider to model_builder format for ModelOpt examples (NVIDIA#2107) * wandb Inference stats logging (NVIDIA#2026) Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> * Make `PipelineParallelLayout` always return str from ` __repr__` (NVIDIA#2055) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Add flash_attn_3 as first option for FA3 import (NVIDIA#2010) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Add debugging hint for case when cudagraphs are created but no matching runner is found (NVIDIA#2129) * ci: LTS container (NVIDIA#2133) Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "ci: LTS container (NVIDIA#2133)" This reverts commit eb48e81. * Fix param init (NVIDIA#2033) Signed-off-by: Chen Cui <chcui@nvidia.com> * Hotfix to unit tests on hopper FA3 (NVIDIA#2143) * Add BytesIO to safe_globals (NVIDIA#2074) * add deprecation warning for legacy tokenizer system (NVIDIA#2145) Signed-off-by: dimapihtar <dpihtar@gmail.com> * replay: ci: Bump LTS container (NVIDIA#2157) Signed-off-by: oliver könig <okoenig@nvidia.com> * Hotfix to unit tests on hopper FA3 (bis) (NVIDIA#2179) * Fix has_modelopt_state() for native Torch checkpoint format (NVIDIA#2160) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * chore: Remove codeowners (NVIDIA#2175) Signed-off-by: oliver könig <okoenig@nvidia.com> * Fix FP8 inference with sequence parallelism (NVIDIA#2009) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> * Replace ModelOpt generation server (NVIDIA#2147) Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> * Add hybrid model support for dynamic inference engine (NVIDIA#1907) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Async task and event loop safety in Megatron Core (NVIDIA#2025) Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> * Rename skip_prompt_log_probs (NVIDIA#2181) * Dynamic inference context | UVM only. (NVIDIA#1983) Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> * Update copy-pr-bot.yaml [skip ci] Signed-off-by: oliver könig <okoenig@nvidia.com> * Revert "Dynamic inference context | UVM only. (NVIDIA#1983)" This reverts commit d6979d6. * ci: Run `auto-update-copy-pr-bot` only on forks (NVIDIA#2191) Signed-off-by: oliver könig <okoenig@nvidia.com> * Inference throughput tests: refactor goldens to be in list format (NVIDIA#2072) * Enable TE custom quantization recipe (NVIDIA#2005) Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: root <Evgeny> * Add MoE parameters to ModelOpt pruning example + conf fixes (NVIDIA#2205) Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Add repr to pg collection class (NVIDIA#2089) Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com> * Move `data_samplers.py` from `legacy` to `training.datasets` & add `DistributedSignalHandler` to DataLoader workers (NVIDIA#2068) * Fix Megatron-FSDP checkpoint save failure (NVIDIA#2138) --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Evgeny <etsykunov@nvidia.com> Signed-off-by: root <Evgeny> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com> Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com> Co-authored-by: helen ngo <helenn@nvidia.com> Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: kanz-nv <kanz@nvidia.com> Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com> Co-authored-by: root <root@gpu-h100-0058.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0220.cm.cluster> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com> Co-authored-by: Evgeny Tsykunov <e.tsykunov@gmail.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
cspades
pushed a commit
to cspades/Megatron-LM
that referenced
this pull request
Jan 7, 2026
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
yangbofun
pushed a commit
to xlm-research/Megatron-LM
that referenced
this pull request
May 22, 2026
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
terminator123
pushed a commit
to 021ai/Megatron-LM
that referenced
this pull request
Aug 3, 2026
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.