chore: nightly sync main into dev (28_07_2026)

[svcnvidia-nemo-ci]

Nightly sync: main -> dev (125 commits)

Automated nightly merge of origin/main into dev for 28_07_2026.

Summary

• Merged 125 commits from main not yet on dev.
• Python lines: +38898 / -5009 across 294 files (excludes golden-value JSON, uv.lock, docs images).
• 369 total files changed vs origin/dev.
• Merge strategy: git merge origin/main --no-edit starting from origin/dev; conflicts resolved surgically (no global -X theirs).

Main features folded in this cycle include: GTP weight-rematerialization, the async-inference scheduling rewrite (#5939, AsyncScheduleMode LEGACY/ASYNC -- SERIAL/ROUND_ROBIN removed), MSC checkpoint I/O, GPT<->hybrid checkpoint interop, per-DP sampling seeds, load-aware prefix routing, CG-admission gating, QKNorm for absorbed MLA (#4452), the RL submission-gate refactor (#5853), and the GDN/GDN2 common-logic split (#5843).

Conflict resolution highlights

• absorbed_mla.py (+ its unit test) -- disjoint-divergence union. Main added QKNorm (Port Multi-Latent Attention to HybridModel #4452) and dev added split-KV-layout ([Dev] Migrate main-first DSA support to dev #6020) on a shared DSA core. Resolved with a 3-way git merge-file union preserving both features (rather than taking either side wholesale, which would have silently dropped dev [Dev] Migrate main-first DSA support to dev #6020 and broken three dev test files). pylint 10.00/10, black/isort clean.
• gated_delta_net.py -- kept dev's evolved single file. Main refactor: extract and split common logic between GDN & GDN2 #5843 split it into a package to share logic with GDN2; this dev tree has no GDN2 consumer and no substructure import, and dev's single file (+518/-115 CP features) satisfies every importer. Main's 3-file package dropped as unused here.
• Override-list files (training/training.py, training/initialize.py, training/datasets/data_samplers.py, core/optimizer/layer_wise_optimizer.py) resolved toward main per skill convention; API-mismatch audit clean.

Protected files (unchanged from dev)

.github/CODEOWNERS, pyproject.toml, uv.lock, docker/Dockerfile.ci.dev are all byte-identical to origin/dev. No git-source reconciliation needed this cycle.

---

Pre-push advisory findings — disposition

Protected files: .github/CODEOWNERS, pyproject.toml, uv.lock, and
docker/Dockerfile.ci.dev are all byte-identical to origin/dev (verified with
git diff --quiet origin/dev -- <file>). No git-source reconciliation was needed
this cycle (main added no new [tool.uv.sources] entries absent from dev).

Merge accident found and fixed — absorbed_mla.py (dev #6020 × main #4452):
This file diverged disjointly: main added QKNorm support (QKNormConfigResolver,
layer_classes) via #4452, while dev added split-KV-layout support
(linear_k_up_proj/linear_v_up_proj, _uses_combined_kv_up_projection) via
#6020. A naive resolution would have dropped one side. Resolved with a 3-way
git merge-file union preserving BOTH features; the single genuine conflict
(the K/V up-projection constructor branch) was resolved by hand to source the
module class from main's layer_classes while keeping dev's split-projection
kv_up_proj_config path. The companion test
tests/.../test_absorbed_mla.py was re-unioned the same way: dev's
dual-layout test_functionality parametrization kept, main's self-contained
_restore_packed_thd_batch_dim / _apply_absorbed_v_up_projection unit tests
re-added, and two of main's tests dropped as redundant with (and, for the
split-KV state-dict test, incompatible with) dev's superior
test_dsa_integration.py coverage. Both files: no conflict markers, syntax OK,
black+isort clean, pylint 10.00/10.

Dev-feature preservation audit — remaining findings all classified class (a)
intentional main removal, or false positive. Symbol-level audit (def/class
names, present on dev, absent on main, absent on merged HEAD). Decisive test per
file: git diff --quiet <merge-base> origin/dev -- <file> (dev==base ⇒ main's
post-branch evolution legitimately owns the file). merge-base(dev,main) =
48a887fecd01674346f724dcf44f417f455989f0.

17 files were dev==base (dev never touched them since the branch point, so
main's removals apply cleanly): examples/mimo/training/step.py,
.../megatron_fsdp/experimental/module.py (DelayedRelease / drain_delayed_releases
/ enqueue_release / release_unsharded_storage),
.../megatron_fsdp/experimental/parameter_group.py (contained_in_parameter_group;
reduce_gradients method removed but class retained — see false positives),
megatron/core/inference/headers.py (false positive — see below),
megatron/core/msc_utils.py (open_file — inlined; no dangling importer),
megatron/core/resharding/planner.py (_extract_metadata; 0 tree-wide refs),
megatron/core/ssm/mamba_layer.py (false positive — see below),
tests/test_utils/python_scripts/notify.py (get_jobs_per_bridge; 0 refs),
tests/unit_tests/a2a_overlap/test_cuda_graphed_schedule_chunk_1f1b.py,
tests/.../mfsdp_v2/test_fully_shard.py,
tests/.../mfsdp_v2/test_symmetric_memory.py,
tests/.../inference/engines/test_dynamic_engine_async_sched.py,
tests/.../inference/test_data_parallel_inference_coordinator.py,
tests/.../inference/test_dynamic_prefix_caching_coordinator.py,
tests/.../models/mimo/test_mimo_1f1b_schedule.py,
tests/.../rl/test_rl_utils.py, tests/.../ssm/ops/test_ssd_combined.py.
The async-sched round-robin/serial test clusters
(test_*_uses_round_robin, test_validate_async_sched_support_for_*,
test_async_sched_serial_step*) are coherent consequences of main #5939's
AsyncScheduleMode enum rewrite (removed SERIAL/ROUND_ROBIN → ASYNC/LOAD_BALANCED).

8 files were dev CHANGED (dev touched the file, but NOT at the removed symbol —
every removed symbol was already defined at the merge-base, i.e. pre-existing code
main deleted, not a dev invention). Verified base_defines=1 for all, and zero
dangling references tree-wide:

• megatron/core/inference/engines/dynamic_engine.py —
  _validate_async_sched_support_for_request: removed by main Extend dynamic inference asynchronous scheduling support #5939 (the async-sched
  enum rewrite; the old body gated on the deleted AsyncScheduleMode.SERIAL). 0 prod refs.
• megatron/rl/agent/api.py — release_after / ReleaseState: renamed to
  release_for(SubmissionGranularity) by main commit af3d24027 (rl: release G-submission gate slots on consumption instead of assembly #5853). Merged
  file uses release_for at all callsites; 0 refs to the old API. merged≠main is
  pure black line-wrapping.
• text_generation_controller.py — DecodeForwardPrimer, _dummy_serial_mtp_forward,
  mark_primed, _run_async_sched_serial_step: all base-defined, removed by main's
  async-sched serial-step rewrite; 0 dangling callers tree-wide.
• chat_completions.py — _reconstruct_reasoning_content: base-defined, removed by main;
  0 refs.
• reward_only_agent.py — _rollout_from_response: base-defined, removed by main; 0 refs.
• megatron/training/checkpointing.py — isfile: base had a module-level
  def isfile() wrapper; main's MSC refactor inlined all 8 callsites to
  maybe_msc.os.path.isfile(...). No dangling call to the removed helper.
• tests/.../test_text_generation_controller.py — helper/test symbols
  (compact_logits, forward_step, _set_nested_attr, test_async_sched_serial_step*,
  test_run_async_sched_forward_records_primer): base-defined, removed alongside the
  serial-step production rewrite.
• tests/.../models/test_hybrid_model.py — test_no_qk_norm_by_default: base-defined,
  removed by main.

False positives (symbol survived in the merged tree; audit flagged cosmetic
line diffs):

• megatron/core/inference/headers.py Headers — still defined at
  headers.py:137 as Headers = _Headers() (a singleton assignment, not a
  class/def, so the def/class grep missed it). Category IntEnums intact.
• megatron/core/ssm/mamba_layer.py LayerNormBuilder — main refactored the
  local class into a shared Protocol in megatron/core/transformer/torch_norm.py;
  merged mamba_layer.py imports it (line 24) and uses it as a type annotation.
  __call__ likewise moved with the refactor.
• examples/mimo/training/step.py move_batch_to_cuda — main relocated it to the
  new examples/mimo/training/batch.py:33.
• .../experimental/parameter_group.py reduce_gradients — the 3 tree-wide refs
  target the unrelated param_and_grad_buffer.py / grad_reduce_pipeline method;
  the removed FsdpParameterGroup.reduce_gradients has no remaining callers.

Docs note: AGENTS.md line about committing with -s/-S was flagged by the
line-level audit — it is identical content sorted differently (false positive).

Files deleted-in-dev vs. new-in-main (restore analysis)

Comparing git ls-tree origin/main vs merged HEAD surfaced 8 paths in main but
not in the merged tree. All correctly resolved:

Not restored — dev intentionally deleted (base=Y, dev=N):

• .github/workflows/multi-approval-bot.yml — removed by dev ci: Remove multi-approval action from dev branch #3576
  ("Remove multi-approval action from dev branch"). Dev governance choice.
• tests/test_utils/recipes/h100/bert.yaml, .../t5.yaml — removed by dev's
  legacy-scope-name changes. Intentional.

Not restored — dead on this dev (base=N, dev=N, new-in-main, zero importers):

• megatron/core/models/common/fine_grained_callables.py and
  megatron/core/models/common/utils.py — main created these as part of the
  in-progress [feat] Hybrid model ep overlapping main #4798 refactor ("Common combined-1F1B schedule-plan base", 1/4).
  Every symbol they would export (build_layer_callables,
  build_mtp_layer_callables, PreProcessNode, PostProcessNode,
  TransformerLayerNode, _BackwardDWWrapper, should_free_input) is defined
  in and imported from megatron/core/models/gpt/fine_grained_callables.py,
  which IS present in the merged tree. No merged-tree code imports the common/
  versions (verified: no absolute import, no common/__init__.py re-export).
  Restoring them would add unimported dead modules. compileall megatron/
  passes clean.

Kept dev's version (disjoint divergence, main's refactor has no consumer here):

• megatron/core/ssm/gated_delta_net.py — dev evolved the single-file module
  (+518/-115 since base: CP/split-section features), while main refactor: extract and split common logic between GDN & GDN2 #5843 split it
  into a gated_delta_net/ package (common.py + gdn.py) to share logic with
  GDN2. This dev tree has NO GDN2 consumer and NO import of the .common/.gdn
  substructure. Dev's single file defines every symbol its importers need
  (GatedDeltaNet, GatedDeltaNetSubmodules, _build_head_perm_for_split_sections,
  _build_thd_cp_a2a_perm, tensor_a2a_cp2hp/hp2cp, and re-exports
  _split_tensor_factory from megatron.core.ssm.utils). Keeping dev's
  more-evolved single file is correct; main's package split (3 files) is dropped
  as it has no consumer on dev. (Contrast with absorbed_mla, where main's feature
  DID need preserving — there the fix was a union; here dev's side is complete.)

Files taken from main (override list)

megatron/training/{training.py, initialize.py},
megatron/training/datasets/data_samplers.py,
megatron/core/optimizer/layer_wise_optimizer.py — resolved toward main per the
skill's override list (they reference args/APIs main removed or renamed).
megatron/training/utils is a package on both dev and main (the skill's
utils.py label is legacy); the merged tree keeps the package form. After taking
main's versions, the API-mismatch audit (compileall megatron/ clean, no dangling
imports) passed.

Verification summary

• 0 conflict markers tree-wide.
• .github/CODEOWNERS, pyproject.toml, uv.lock, docker/Dockerfile.ci.dev
  byte-identical to origin/dev.
• python3 -m compileall megatron/ → exit 0 (only a pre-existing docstring
  SyntaxWarning in kv_block_allocator.py, unrelated to the merge).
• absorbed_mla.py + its test: black/isort clean, pylint 10.00/10.

---

Remerge-diff (conflict resolutions only, 12121 lines / 804 KB) -- reproduce locally due to size

The full git show --remerge-diff HEAD output is 12k lines / 804 KB (too large to inline). It covers 71 conflicted paths. Reviewers can reproduce it with:

git fetch origin main2dev/28_07_2026
git show --remerge-diff origin/main2dev/28_07_2026

Key conflict-resolved production files: absorbed_mla.py, dynamic_engine.py, text_generation_controller.py, api.py, transformer_layer.py, token_dispatcher.py, router.py, experts.py, optimizer.py, transformer_config.py, arguments.py, training.py, initialize.py, checkpointing.py, gated_delta_net.py, hybrid_block.py, model_chunk_schedule_plan.py, pretrain_gpt.py, pretrain_hybrid.py.

Automated nightly sync. Generated with Claude Code.

[svcnvidia-nemo-ci]

/ok to test da1aa99

[svcnvidia-nemo-ci]

Phase-3 fix round 2 — SHA da1aa9930

Folded into the single rolling fix commit (amended on top of the immutable merge 8fd9908c4; two-commit structure preserved). Root-caused the 6 remaining unit-test buckets from the prior run.

Sync-caused code fixes (override → callee/default mismatches):

Bucket	Root cause	Fix
data/**	Merge took main's get_data_parallel_world_size signature (new with_gtp_remat=True default). Dev-only get_dynamic_data_context_parallel_groups then read the GTP_remat DP-with-CP group, which is None during initialize_model_parallel (GTP_remat groups are built later).	In the dev function, query get_data_parallel_world_size(..., with_gtp_remat=False) — matches the replicate _DATA_PARALLEL_GROUP_WITH_CP the branch returns and the plain DP×CP group_size.
transformer/moe/**	Dev scales the activation by aux_loss_scale_num_tokens (= total_num_tokens); the merge grafted main's valid_token_count=… * bsz, but that arg is dead when aux_loss_scale_num_tokens is non-None. bsz was folded into the expert dim, so total_num_tokens omits the batch factor → MBS=N grad = 1/bsz of MBS=1 (grad_mbs1 == 4× grad_mbsN).	aux_loss_scale_num_tokens=total_num_tokens * bsz (equals main's valid_token_count * tp_cp_group.size()); revert valid_token_count to dev's local_num_tokens.
catchall (argument_utils)	gpt_config_from_args/hybrid_config_from_args used main's def-time default model_config_cls=GPTModelConfig + unconditional assert issubclass(...), which broke dev's tests that monkeypatch the module-level class to a MagicMock.	Resolve the default at call time (= None → resolve to GPTModelConfig/HybridModelConfig); only assert issubclass for an explicitly-supplied class. Preserves main's ModelOpt override.
inference (dynamic_context)	Main-only test test_hybrid_prefix_caching_without_mamba_budget_warns survived the merge, but the merge dropped main's elif branch that emits the warning.	Restore the memory-only elif self.is_hybrid_model and self.config.enable_prefix_caching: warning branch.

Main-only tests gated (backend absent from dev's mandated container):

Both files are HEAD == main verbatim (dev = 0 lines), added by this sync, and green on main's own CI (scheduled run 2026-07-29, SHA 157c023f2f: transformer/** - latest ✅ and mfsdp_v2/** - latest ✅). The dev container keeps its own base image / dependency triple (mandated — not to be changed), which lacks the backends these tests need. Fixed with graceful skips matching each bucket's existing skip idiom, not code changes:

• test_dsa_tilelang_kernels.py::test_lighting_indexer_indices_preserves_single_head_weight_axis — skip when indexer.lighting_indexer_indices is None (TileLang indexer kernels unavailable). Its sibling real-kernel tests already skip via _skip_if_real_tilelang_indexer_unavailable(); this one omitted the guard and raised TypeError: 'NoneType' object is not callable.
• mfsdp_v2/profiler_utils.py::collect_linked_kernels (single chokepoint for all profiler-overlap tests) — skip when the running torch's profiler FunctionEvent lacks linked_correlation_id (added in a newer torch than dev's base image ships), instead of raising AttributeError.

DCO disposition: the required DCO check reports ACTION_REQUIRED for 50 inherited upstream commits (authored by various NVIDIA devs; author/committer email mismatches from GitHub squash-merges) — not commits created by this sync. The only two sync-authored commits (merge 8fd9908c4, fix da1aa9930) are correctly signed off. Fixing this would require rebasing/rewriting ~127 commits of others' history, which is forbidden and destructive. This is the identical pre-existing condition on prior sync PRs (#5716, #5627) and on the last merged sync PR #5784.

Advisory pre-push audit: CODEOWNERS and the dependency triple (pyproject.toml, uv.lock, docker/Dockerfile.ci.dev) are byte-identical to dev. The dev-feature audit's line-level findings are the whole-merge baseline already dispositioned in the PR body; this round's removals are the four deliberate fixes above.

[svcnvidia-nemo-ci]

/ok to test 0ba8a2a

[FDecaYed]

Potential issues — owner confirmation requested; no fix is proposed here:

1. MTP block-scope CUDA graph state/buffer — @santhnm2 (#4950)

   The chore: nightly sync main into dev (28_07_2026) #6112 result does not contain the MTP CUDA-graph buffer state and allocation present in main after Add full model cuda graph support for MTP inference #4950 (current, main). Could you please take a look and confirm the intended resolution?

2. Compact LayerWise/Muon DDP wiring — @Wohox (#5388, #5470)

   The LayerWise DDP setup in chore: nightly sync main into dev (28_07_2026) #6112 is gated by use_layer_wise_param_layout (current), which differs from the dev-side layout setup (reference). Could you please take a look and confirm the intended resolution?

3. Generic combined-1F1B schedule-plan hooks and mHC work — @Connor-XY @jingqiny-99 (#4941, #5471)

   chore: nightly sync main into dev (28_07_2026) #6112 retains GPT-specific schedule-plan callable wiring (current), while the generic hooks and common callable package from [refactor] Common combined-1F1B schedule-plan base (1/4 of #4798) #4941 are absent (main reference). Could you please take a look and confirm the intended resolution?

4. GDN/GDN2 common-code split — @xuantengh @yuzhongw-nvidia (#5843, WIP #6088)

   chore: nightly sync main into dev (28_07_2026) #6112 retains the monolithic gated_delta_net.py, while refactor: extract and split common logic between GDN & GDN2 #5843 uses the package split. [dev] Refactor: extract and split common logic between GDN & GDN2 (cherry-pick #5843) #6088 is the WIP dev-side work for this conflict. Could you please confirm whether this sync should drop the change or pull in the work from [dev] Refactor: extract and split common logic between GDN & GDN2 (cherry-pick #5843) #6088?

[xuantengh]

I think this sync can pull the GDN split, this is a relatively separated change and we also keep the compatibility.

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.