chore: nightly sync main into dev (13_07_2026) by svcnvidia-nemo-ci · Pull Request #5784 · NVIDIA/Megatron-LM

svcnvidia-nemo-ci · 2026-07-13T18:49:14Z

Summary

Nightly sync of main into dev for 13_07_2026. Merges 116 commits from main into dev.

Python lines: +36836 / -4228 across 278 files (includes main's 116 commits; conflict-resolution surface is far smaller — see remerge-diff below).

91 files had merge conflicts, all resolved surgically per the nightly-sync skill (preserve dev-only additions; take main only for the sanctioned override list or where main's refactor is authoritative).

Key resolution decisions

Dependency triple & environment (kept dev's):

pyproject.toml, uv.lock, docker/Dockerfile.ci.dev — kept dev's verbatim (verified identical to origin/dev).
docker/.ngc_version.dev + .gitlab/stages/01.build.yml — kept dev's 26.04 base image. Main bumped to 26.06 via Update base image to nvcr.io/nvidia/pytorch:26.06-py3 #5632, a dedicated base-image-bump PR that bundled regenerated golden values + GitLab pin. Base-image bumps belong in their own PR (cf dev's own chore: Update Docker image version to 26.04-py3 on dev #5051), not the nightly sync, and cannot be validated here (no GPU / uv.lock regen). Golden values (19 JSONs) kept as dev's for the same reason.
.github/CODEOWNERS — dev's verbatim (never modified; verified empty diff).
.github/copy-pr-bot.yaml — union of trustee lists (kept dev-only sanandaraj5597, wplf; added main's new trustees).

API-mismatch fixes (main callers → dev's renamed APIs):

args.hybrid_context_parallel → args.dynamic_context_parallel (training.py, pretrain_hybrid_flex.py) — the deprecated hybrid arg does not exist in the merged argparse.
get_hybrid_data_context_parallel_groups → get_dynamic_data_context_parallel_groups (training.py import + call site) — only the dynamic variant exists in merged parallel_state.

Major subsystem merges:

DSA/DSv4: restored dev's coherent MLASelfAttention + DSAttention stack after CI exposed an incompatible mix of main's absorbed/fused DSA path and dev's callers. Main-only fused-backend modules, tests, config fields, and dependency helpers were removed together; dev's separate apply_dsa_kernel_fusion path remains. Porting main's [2/2] Wiring cuDNN fused DSA kernels support with THD, CP and IndexShare (GLM5.2) #5099/[split 4/4] Enable DSA CP and THD hooks #5246 backend onto dev's DeepSeek-v4 architecture is deferred to a dedicated follow-up.
MoE flex dispatcher (transformer_config.py, token_dispatcher.py): unioned backend Literal (deepep/deepepv2/hybridep/ncclep); kept both _DeepepV2Manager (dev) and _NCCLEPManager (main); took main's unified moe_flex_dispatcher_num_sms (with deprecated moe_deepep_num_sms routed); preserved dev's DSv4/CSA __post_init__ validation and fine_grained_offloading_max_inflight_offloads.
transformer_layer.py: kept dev's _forward_mlp_output_with_bias / input_ids router plumbing and main's inter-document-masking MoE reshape (_maybe_unflatten_for_moe/_maybe_reflatten_from_moe).
get_batch: pretrain_gpt.py kept dev's implementation (matches the merged training.utils helper world) + grafted main's cached-logits KD loss_func branch. pretrain_hybrid.py/pretrain_hybrid_flex.py use main's core.utils helper signatures with dev's dynamic CP-group naming.
MTP inference: kept dev's _decoder_hidden_states_cache path (consumed by text_generation_controller); dropped main's block-scope inference_context.mtp_decoder_hidden_states approach (its infra is not present in the merged tree). Dropped main's TestMTPBlockScopeCudaGraph tests accordingly.

Files taken wholly or primarily from main (their refactors are authoritative): megatron/training/training.py (main overall, with dev's final DSv4/MLA FLOPs estimator region restored after CI), megatron/training/checkpointing.py (process-group threading — callers already pass dp_group/dp_cp_group/rng_state_key_prefix), .github/scripts/oncall_manager.py, .github/scripts/sync_team_usergroups.py (Slack helpers extracted to github_slack_utils), examples/post_training/modelopt/finetune.py, tools/trigger_internal_ci.py, megatron/inference/utils.py, megatron/core/pipeline_parallel/fine_grained_activation_offload.py (main's group_offload/should_bulk_offload(group) interface — the merged common regions already require it).

mFSDP test reorg: main renamed tests/.../megatron_fsdp/ → mfsdp_v1/+mfsdp_v2/ (#5640). Dev's new test_mfsdp_param_and_grad_buffer.py (dev-only #5013) preserved under mfsdp_v1/.

Intentional main-side removal (documented per skill case (a)): _forward_mlp_postprocess's redundant _restore_token_dispatcher_attrs()-at-top was removed by main commit 5e4fe9b3c ("Optimize memory usage of partial CUDA graphs"), replaced by its weakref tail.

Verification performed

All changed .py files compile (py_compile).
black==24.4.2 (--skip-magic-trailing-comma --skip-string-normalization, line-length 100) + isort==5.13.2 (pyproject config) pass cleanly on the 323-file set CI lints (megatron/core + tests/ changed vs main). Note: main's training.py/checkpointing.py were not black-v24-clean; dev's standard is, so resolved files were reformatted to dev's standard (what CI enforces).
CODEOWNERS and dependency triple verified byte-identical to origin/dev.
No dangling imports/calls of any main-removed symbol. The legacy RL parallel-generation flags are intentionally superseded by main's submission/consumption-granularity API from a58373f33 (Add RL rollout submission and consumption granularity controls #5306).

⚠️ Note on the pre-push dev-feature hook

The exact-line heuristic flagged 2,468 lines / 192 files on the raw merge and 1,925 / 186 on the amended fix. Contrary to the initial note, four signals were genuine merge accidents: dev's DSA pass validation and three MLASelfAttention test lines. Those are restored. CI also found two blind spots the heuristic cannot see: the exempt training.py FLOPs region and a main-origin MTP import; both are fixed.

A second split audit traced every remaining high-signal def/class/import/call/config-field removal to the target implementation and its specific main refactor. Key evidence: RL granularity/pipeline/registry a58373f33, 027fa4aab, 48a887fec; CUDA graph/offload 69c486825, 5e4fe9b3c; MIMO 052209940, f8170b443, e1c449557; dist-checkpointing and PG threading a2496aa17, 40b1fd37c, 903838132, 817c1d5e6, 2a21b8eb9; FSDP da42015c8, f285ea5fa, adfe9e11d, 25f61179c, 47cb41364, 72a78d62c; SSM helper deduplication 872442adc; CP/data helpers 8fa183119, 2a43e0d65, f88b85f8c, bf32f4415; inference 4d44e37b7, a27b04024, 905ba419b, a2bb5e543, 3c327f373; MoE/NCCL-EP 2d7060f44, 115ce7ffd; ModelOpt/training migration 223e244f5, 2b551c687, 811bd2946; GitHub helper extraction 3330d12b9; determinism 3ab71eeb3. No further dangling import, signature, config field, production definition, or call site remains. CODEOWNERS and the dependency triple are byte-identical to the dev parent.

remerge-diff file summary (conflict resolutions only, 91 files)

 .github/CODEOWNERS                                 |   65 --
 .github/copy-pr-bot.yaml                           |    6 +-
 .github/scripts/oncall_manager.py                  |  137 ---
 .github/scripts/sync_team_usergroups.py            |  115 --
 .gitlab/stages/01.build.yml                        |    4 +-
 docker/.ngc_version.dev                            |    6 +-
 docker/Dockerfile.ci.dev                           |   13 -
 .../features/fine_grained_activation_offloading.md |    6 +-
 examples/bert/pretrain_bert.py                     |    8 +-
 .../inference/advanced/gpt_dynamic_inference.py    |    5 -
 .../gpt_dynamic_inference_with_coordinator.py      |    3 -
 examples/post_training/modelopt/finetune.py        |  126 +--
 gpt_builders.py                                    |   10 -
 megatron/core/datasets/gpt_dataset.py              |    4 +-
 .../fsdp/src/megatron_fsdp/megatron_fsdp.py        |    3 -
 .../src/megatron_fsdp/param_and_grad_buffer.py     |    3 -
 megatron/core/extensions/transformer_engine.py     |   11 +-
 megatron/core/inference/engines/dynamic_engine.py  |    7 -
 .../experimental_attention_variant_module_specs.py |   10 +-
 megatron/core/models/gpt/gpt_model.py              |   14 +-
 megatron/core/models/hybrid/hybrid_block.py        |    5 +-
 megatron/core/models/hybrid/hybrid_model.py        |   18 -
 megatron/core/packed_seq_params.py                 |    3 -
 .../fine_grained_activation_offload.py             |   67 --
 megatron/core/pipeline_parallel/schedules.py       |   25 -
 megatron/core/ssm/gated_delta_net.py               |    3 -
 megatron/core/transformer/attention.py             |   14 +-
 megatron/core/transformer/cuda_graphs.py           |   22 -
 .../experimental_attention_variant/dsa.py          |  909 +++-------------
 .../experimental_attention_variant/dsa_kernels.py  |   51 +-
 megatron/core/transformer/module.py                |    3 -
 megatron/core/transformer/moe/experts.py           |   11 -
 megatron/core/transformer/moe/token_dispatcher.py  |  174 ++-
 .../core/transformer/multi_latent_attention.py     |    3 -
 megatron/core/transformer/transformer_block.py     |    5 +-
 megatron/core/transformer/transformer_config.py    |   85 +-
 megatron/core/transformer/transformer_layer.py     |  118 +-
 megatron/core/utils.py                             |   16 -
 megatron/elastification/pretrain_hybrid_flex.py    |  101 +-
 megatron/inference/utils.py                        |   36 +-
 megatron/post_training/checkpointing.py            |   13 +-
 megatron/rl/agent/api.py                           |  103 +-
 megatron/rl/agent/reward_only_agent.py             |   31 +-
 megatron/rl/inference/megatron.py                  |    4 -
 megatron/rl/rl_utils.py                            |   70 +-
 megatron/training/argument_utils.py                |   16 +-
 megatron/training/arguments.py                     |  817 ++++----------
 megatron/training/checkpointing.py                 |  240 ++---
 megatron/training/config/__init__.py               |   10 +-
 megatron/training/config/container.py              |   11 +-
 megatron/training/distillation/utils_logits.py     |    5 -
 megatron/training/models/gpt.py                    |   14 +-
 megatron/training/models/hybrid.py                 |    4 -
 megatron/training/training.py                      |  865 ++-------------
 megatron/training/utils/common_utils.py            |   10 +-
 megatron/training/yaml_arguments.py                |   12 +-
 pretrain_gpt.py                                    |  148 +--
 pretrain_hybrid.py                                 |   30 +-
 pyproject.toml                                     |    7 +-
 .../golden_values_dev_dgx_gb200.json               |  858 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  664 +++---------
 .../golden_values_dev_dgx_gb200.json               |  870 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  448 ++++----
 .../golden_values_dev_dgx_gb200.json               |  767 +++++--------
 .../golden_values_dev_dgx_gb200.json               |  866 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  858 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  387 +++----
 .../golden_values_dev_dgx_gb200.json               |  387 +++----
 .../golden_values_dev_dgx_gb200.json               |  401 +++----
 .../golden_values_dev_dgx_gb200.json               |  448 ++++----
 .../golden_values_dev_dgx_gb200.json               |  428 ++++----
 .../golden_values_dev_dgx_gb200.json               |  381 +++----
 .../golden_values_dev_dgx_gb200.json               |  446 ++++----
 .../golden_values_dev_dgx_gb200.json               |  206 +---
 .../golden_values_dev_dgx_gb200.json               |  894 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  896 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  896 +++++++--------
 .../golden_values_dev_dgx_gb200.json               |  886 +++++----------
 .../a2a_overlap/test_fsdp_1f1b_overlap.py          |   11 -
 .../inference/test_mtp_cuda_graph_inference.py     |  371 -------
 tests/unit_tests/models/test_hybrid_moe_model.py   |    3 -
 .../test_fine_grained_activation_offloading.py     |   61 --
 tests/unit_tests/ssm/test_hybrid_block.py          |    5 +-
 .../test_absorbed_mla.py                           |  114 --
 .../test_attention_variant_dsa.py                  |   22 +-
 .../transformer/moe/test_token_dispatcher.py       |   32 +-
 tools/bert_embedding/embed.py                      |    8 +-
 tools/run_inference_performance_test.py            |    4 -
 tools/trigger_internal_ci.py                       |   46 +-
 train_rl.py                                        |   24 +-
 uv.lock                                            | 1139 +-------------------

🤖 Generated with Claude Code

Latest CI follow-up

Workflow 29349138534 confirmed the broad transformer, RL, SSM, all-unit, MBridge, and broad MoE suites pass on the rolling fix. Its dedicated MTP failure was a stale test call using the pre-223e244f5 setup_model_and_optimizer argument order; the test now passes ModelType first and model_provider second.

The A2A-overlap job had one numerical mismatch in the HybridEP padding-mask case. The exact synthetic merge tested already contains dev fix d1384c2d9 and its dedicated regression tests, and the same case passed on that fix and on the prior nightly SHA, so no duplicate production change was added; the case is being rerun unchanged on the amended SHA.

CI follow-up: run 29383265299

Workflow 29383265299 ran on e348ed156e84b171387fa42a6a7a1b6a1153439b. All unit-test buckets passed, including the previously failing A2A-overlap case. The non-MBridge functional failures had exactly two root causes across all 88 failed artifacts:

87 GPT/MoE jobs: the merged tree kept dev's positional pretrain(..., model_provider, model_type, forward_step) call order while main commit 223e244f5 changed the API to pretrain(..., model_type, forward_step, model_provider=None). This bound forward_step as the model provider and failed with TypeError: forward_step() got an unexpected keyword argument 'pre_process'.
Hybrid VPP job 87268231028: the merge adopted 223e244f5's config-container ModelBuilder path and removed the obsolete “VPP must raise” unit test, but accidentally retained dev's stale runtime assertion. Core HybridModel already accepts, stores, and uses vp_stage.

Rolling fix dc693dae557174b95da76b2c7e3ab7d6d832b7c7 aligns the GPT and multimodal call sites with the new signature and restores main's intentional Hybrid VPP assertion removal. The pre-push dev-feature audit adds exactly seven removed exact lines versus the prior PR head (two obsolete positional-provider lines and five assertion lines); all seven are intentional consequences of 223e244f5, not dropped dev functionality.

Local verification: Ruff and py_compile pass on all three changed files; the Hybrid builder unit file passes in the existing Megatron CI container (35 passed). MBridge is report-only for this iteration per the PR owner's direction.

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Signed-off-by: wdykas <wdykas@nvidia.com>

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Laura Dang <laurad@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>

Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com> Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com> Co-authored-by: shanmugamr1992 <shanmugamr1992@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Signed-off-by: Jingyue Wu <wujingyue@gmail.com>

…5326)

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Signed-off-by: Jingyue Wu <wujingyue@gmail.com>

…and remove legacy modelbuilder functions (#5169) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Hollow Man <hollowman@opensuse.org>

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: yanghao.666 <yanghao.666@jd.com>

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

…ls) (#5469) Signed-off-by: ilml <tolong@nvidia.com>

Signed-off-by: ykarnati <ykarnati@nvidia.com>

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Phlip79 · 2026-07-15T02:05:52Z

/ok to test e348ed1

Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

Phlip79 · 2026-07-15T16:35:09Z

/ok to test dc693da

Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

Phlip79 · 2026-07-15T19:42:10Z

/ok to test 015de62

svcnvidia-nemo-ci · 2026-07-16T15:36:38Z

Superseded by today's nightly sync.

FDecaYed · 2026-07-17T07:40:24Z

/ok to test 48ca9de

# Conflicts: # megatron/training/arguments.py # pretrain_hybrid.py # tests/unit_tests/data/test_get_batch.py

FDecaYed · 2026-07-17T07:45:40Z

/ok to test 775f4e3

FDecaYed

added minor TP fused MLA fix and ncclEP flag change. Also resolved merge conflicts with latest dev.

FDecaYed · 2026-07-17T10:10:12Z

/ok to test 1de26bc

FDecaYed · 2026-07-17T13:08:43Z

/ok to test a77ba35

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Phlip79 · 2026-07-17T20:00:42Z

/ok to test 63754d9

svcnvidia-nemo-ci · 2026-07-20T15:42:29Z

Superseded by today's nightly sync.

yashaswikarnati and others added 30 commits June 22, 2026 19:15

Support the MIMO cross-grid path in training loop (#5373)

e1c4495

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Stabilize hybrid_2b GB200 perf test against run-to-run noise (#5364)

6bd392f

Consistent oncall schedule (#5404)

b6b44a7

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Disag MR3: Add heterogeneous KV/Mamba reshard planners (#5188)

93a7642

Signed-off-by: wdykas <wdykas@nvidia.com>

Add RADIO vision encoder wrapper for MIMO example (#5397)

2a46893

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Clean up MTP inference control flow (#5418)

76f6ccc

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Add MIMO dual gradient finalization (colocated + non-colocated) (#5286)

f8170b4

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add RL rollout submission and consumption granularity controls (#5306)

a58373f

Signed-off-by: Laura Dang <laurad@nvidia.com>

Add --functional-test-name to trigger_internal_ci (#5449)

f66c28f

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rename CP batch helpers to describe balancing granularity (#5403)

8fa1831

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

build: point flash_mla at the nv_dev branch (#5448)

06ae6a9

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add logprobs_mode (raw/processed) to inference config (#5419)

a2bb5e5

Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>

Add hetero grid args and MoE process groups for MIMO example (#5375)

b1884d1

Signed-off-by: ykarnati <ykarnati@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

ci: Set test_save_verify_integrity_manifest_directly as flaky (#5468)

b549290

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Remove DBuffer mesh axis validation (#5441)

47cb413

Signed-off-by: Jingyue Wu <wujingyue@gmail.com>

feat(inference): default use_coordinator to True in high-level APIs (#…

a27b040

…5326)

Support HybridModel feature specs in ModelOpt (#5354)

811bd29

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Add experimental Megatron-FSDP fully_shard implementation (#5387)

e7af860

Signed-off-by: Jingyue Wu <wujingyue@gmail.com>

chore: rotate oncall schedule

cc0c960

Add inference functions to support MCore-/MBridge- training refactor …

4d44e37

…and remove legacy modelbuilder functions (#5169) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ci: launch GB200 unit tests via launch_on_gb200 marker (#5477)

5a256f3

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

build: install flash_mla from source in the CI image (#5481)

82de1b8

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

[split 2/4] Scale DSA indexer loss in pipeline schedules (#5244)

0b0d985

Signed-off-by: Hollow Man <hollowman@opensuse.org>

ci: check megatron.training imports in installation test (#5458)

0938eb7

Signed-off-by: oliver könig <okoenig@nvidia.com>

Fix merges_file kwarg name in HuggingFaceTokenizer (#5406)

239959b

Signed-off-by: yanghao.666 <yanghao.666@jd.com>

Automated community request assignment (#5147)

3330d12

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

Clean up training.py module header (dedupe + reorganize imports/globa…

311416f

…ls) (#5469) Signed-off-by: ilml <tolong@nvidia.com>

Thread process groups through training checkpoint paths (#5486)

9038381

Signed-off-by: ykarnati <ykarnati@nvidia.com>

Narrow oncall responsibilities (#5490)

5863721

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

copy-pr-bot Bot temporarily deployed to public July 15, 2026 02:06 Inactive

copy-pr-bot Bot temporarily deployed to test July 15, 2026 02:07 Inactive

copy-pr-bot Bot temporarily deployed to public July 15, 2026 02:10 Inactive

copy-pr-bot Bot temporarily deployed to public July 15, 2026 02:20 Inactive

fix: align pretrain provider call order

dc693da

Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

copy-pr-bot Bot temporarily deployed to public July 15, 2026 16:35 Inactive

copy-pr-bot Bot temporarily deployed to test July 15, 2026 16:36 Inactive

fix: restore dev training-loop behavior

015de62

Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

Phlip79 approved these changes Jul 16, 2026

View reviewed changes

fix: restore MLA TP fusion and DeepEP v2 SM config

48ca9de

Merge remote-tracking branch 'origin/dev' into main2dev/13_07_2026

775f4e3

# Conflicts: # megatron/training/arguments.py # pretrain_hybrid.py # tests/unit_tests/data/test_get_batch.py

FDecaYed approved these changes Jul 17, 2026

View reviewed changes

fix: address post-merge unit test regressions

1de26bc

fix: resolve callable hybrid specs

a77ba35

fix: preserve static hybrid module specs

63754d9

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>

svcnvidia-nemo-ci mentioned this pull request Jul 29, 2026

chore: nightly sync main into dev (28_07_2026) #6112

Closed

Navigation Menu

Uh oh!

chore: nightly sync main into dev (13_07_2026) - #5784

chore: nightly sync main into dev (13_07_2026)#5784
balasaajay merged 126 commits into
devfrom
main2dev/13_07_2026

svcnvidia-nemo-ci commented Jul 13, 2026 •

edited by Phlip79

Loading

Uh oh!

Phlip79 commented Jul 15, 2026

Uh oh!

Phlip79 commented Jul 15, 2026

Uh oh!

Phlip79 commented Jul 15, 2026

Uh oh!

svcnvidia-nemo-ci commented Jul 16, 2026

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

FDecaYed left a comment

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

Phlip79 commented Jul 17, 2026

Uh oh!

svcnvidia-nemo-ci commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Uh oh!

Conversation

svcnvidia-nemo-ci commented Jul 13, 2026 • edited by Phlip79 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key resolution decisions

Verification performed

⚠️ Note on the pre-push dev-feature hook

Latest CI follow-up

CI follow-up: run 29383265299

Uh oh!

Phlip79 commented Jul 15, 2026

Uh oh!

Phlip79 commented Jul 15, 2026

Uh oh!

Phlip79 commented Jul 15, 2026

Uh oh!

svcnvidia-nemo-ci commented Jul 16, 2026

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

FDecaYed left a comment

Choose a reason for hiding this comment

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

FDecaYed commented Jul 17, 2026

Uh oh!

Phlip79 commented Jul 17, 2026

Uh oh!

svcnvidia-nemo-ci commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

svcnvidia-nemo-ci commented Jul 13, 2026 •

edited by Phlip79

Loading