chore: nightly sync main into dev (13_08_2026)

[svcnvidia-nemo-ci]

Summary

Nightly sync of main into dev.

• 274 commits from main (merge-base 48a887fec).
• Python lines: +69055 / -11351 across 474 files
• 945 files changed in total vs origin/dev (29 golden-value JSONs, uv.lock, recipe YAMLs, etc.).
• Merge performed as git merge origin/main --no-edit from origin/dev. No global -X theirs; every conflict was resolved individually.

Files where main's version was taken

Per the nightly-sync skill's "Files to Override from Main" list:

File	Notes
megatron/training/training.py	Taken from main, plus 4 deliberate changes — see below
megatron/training/utils/common_utils.py	Taken from main (dev's utils.py path is now this module)
megatron/training/datasets/data_samplers.py	No conflict; main's changes applied cleanly
megatron/core/optimizer/layer_wise_optimizer.py	Constructor signature from main
megatron/training/initialize.py	No conflict this cycle

training.py — 4 intentional deviations from main

1–2. args.hybrid_context_parallel → args.dynamic_context_parallel (2 sites: :2407, :3593).
The skill requires this rename after taking main's version. It is a correctness fix, not cosmetic:
model_parallel_config.__post_init__ (megatron/core/model_parallel_config.py:573-584) maps
hybrid_context_parallel → dynamic_context_parallel = True but never the reverse, so main's
reference reads False for every user who passes --dynamic-context-parallel, silently skipping the
HybridCPDataLoaderWrapper wrap and mis-setting is_hybrid_cp.

3–4. Restored _get_thd_sequence_length_upper_bound + its call-site kwarg. This was a genuine
merge accident — details in the advisory-findings section below.

Files explicitly kept at dev's version

.github/CODEOWNERS, pyproject.toml, uv.lock, docker/Dockerfile.ci.dev — all four verify
byte-identical to origin/dev. They appear in the remerge diff only because a conflict was
resolved back to dev's version. uv.lock was never hand-edited and needed no regeneration.

Git source reconciliation: main has two [tool.uv.sources] entries dev lacks (deep_gemm,
mamba-ssm). Neither needs adding: every import of both is try/except ImportError-guarded
(batch_invariant_kernels.py:33, :1740; core/utils.py:398), and mamba-ssm~=2.2 is already a dev
dependency from PyPI (pyproject.toml:95, :119, :179). Three sources differ in revision
(emerging_optimizers, nemo-run, transformer-engine); dev is newer on emerging_optimizers, and
the skill mandates keeping dev's coupled triple, so dev's revisions were kept.

Files deleted in dev but restored

None restored — and that is the correct outcome. Comparing origin/main against the merged tree
surfaced 3 candidates:

File	Decision	Evidence
tests/test_utils/recipes/gpt-nemo.yaml	Left deleted	Dev's 40d30942d "fix: remove files that dev intentionally deleted" removes exactly these 3 files — a prior sync had restored them by mistake
tests/functional_tests/test_cases/gpt/gpt3_nemo_te_tp1_pp1_1node_50steps/model_config.yaml	Left deleted	idem
tests/functional_tests/test_cases/gpt/gpt3_nemo_te_tp1_pp1_1node_50steps/golden_values_dev_dgx_h100.json	idem	idem

Restoring a recipe file is not inert: recipe_parser.py:389 discovers recipes via
recipes_dir.glob("**/*.yaml"), so putting one back silently re-adds whole functional-test suites to
the CI matrix.

Disposition of every pre-push advisory finding

The audit reported 6,486 potential dev-only line removals, plus no CODEOWNERS finding and
no dependency-triple finding.

That 6,486 count is not meaningful as stated: black reformatted 15 files and isort reordered
imports in 3, and the audit's predicate is comm -23 over sorted raw lines, so every rewrapped
line reads as a missing dev line. I re-ran the identical predicate at two granularities that
reformatting cannot perturb:

Granularity	Findings
Raw lines (as the hook reports)	6,486
Identifiers (AST Name/Attribute/keyword/arg/alias)	1,047
Definitions only (def/class)	140 ← high-signal set

Classifying all 140 by whether the name exists anywhere in the merged tree:

• 54 RELOCATED (moved by a main refactor, still present). Verified against the known refactors —
  e.g. 13 schedule-node symbols from models/gpt/fine_grained_callables.py → models/common/utils.py
  and 4 → models/common/fine_grained_callables.py (main's ffbe018c8); 12 pipeline symbols from
  megatron/rl/agent/api.py → megatron/rl/agent/rollout_pipeline.py (main's 87bf52075).
• 86 ABSENT EVERYWHERE, triaged individually:
  • 78 → case (a), main intentionally removed, each with a specific commit and no dev activity
    after the merge-base. Representative: c922805a4 (Add support for non-Gym multi-turn environments #5312) rewrote
    tests/unit_tests/rl/test_rl_utils.py (554 lines changed) and renamed
    test_get_grouped_rollouts.py → test_rollout_generation.py, dropping 11 dev test names.
    c922805a4 is an ancestor of main but not of dev, and dev never touched those tests after
    the merge-base — dev's copies were simply the older pre-rewrite versions. Others: 87bf52075
    (RL: unify rollout generation into one pipeline  #6495), 541d5eef0, f41ec5495, bcf4c8fb5 (inference async-sched/coordinator), 59b72fa57,
    d981f66be, 714c24a7e, d207685b4 (MFSDP v2), 648bc011f, 4464d1c19, 2a75ac12c.
    In every case the merged file is byte-identical to main's and main's version is the larger,
    newer one (test_rl_utils.py 1183 → 1393 lines; test_fully_shard.py 593 → 1083), i.e. main is
    ahead on those paths.
  • 1 → case (b), REAL MERGE ACCIDENT, restored. See below.
  • 7 → verified harmless. _dsv4_hybrid_self_attention_flops, mla_attn_layer_flops,
    finalize_model_grads_with_state_reload are internal to dev's training.py (definition +
    callers all inside the file), which is on the override list. Confirmed no dangling external
    caller: the only surviving mentions of the first two are inside docstrings of
    tests/unit_tests/test_num_floating_point_operations.py (:292, :300, :1019) with no
    executable reference, and that test imports only num_floating_point_operations,
    update_seqlen_stats_from_cu_seqlens, consume_seqlen_stats_in_iteration — all present.
    finalize_model_grads_with_state_reload has zero references tree-wide. DelayedRelease,
    Headers, DecodeForwardPrimer, TransformerLayerState, _RolloutPipeline are classes in
    files where main's version won, with no surviving callers.

The one real merge accident: _get_thd_sequence_length_upper_bound

Exactly the pattern the skill warns about — main's caller version dropped a dev-only feature whose
callee still supports it.

• Dev defined the helper and passed it: TECudaGraphHelper(..., thd_sequence_length_upper_bound=_get_thd_sequence_length_upper_bound(args)).
• Main has neither the helper nor the kwarg.
• But the callee still accepts it: megatron/core/transformer/cuda_graphs.py:2150
  (thd_sequence_length_upper_bound=None), stored at :2166, consumed at :2679-2682.

Impact if left dropped — two distinct failures:

1. Hard test failure. tests/unit_tests/training/test_train_step_schedule_plumbing.py:347 asserts
   training_mod._get_thd_sequence_length_upper_bound(args) == 1024. The merged tree keeps dev's
   copy of that file (347 lines; main's is 69 and never references the helper), so it would raise
   AttributeError. This is sync-caused, not pre-existing — the test passes on dev.
2. Silent correctness bug. Without the kwarg, cuda_graphs.py:2679 falls back to
   self.seq_length, undersizing THD CUDA graphs. Per the comment there, seq_length is wrong under
   CP because max_seqlen_per_dp_cp_rank is only the per-rank token budget, not the max length of one
   sample before packing.

Fixed by restoring dev's 35-line helper verbatim (math already imported) and re-adding the kwarg.
I then wrote an AST checker for every from-import and module-alias attribute reference against all
override-list modules plus data_schedule.py: 0 unresolved.

Golden-value JSONs: a new CI gate arriving from main

tools/check_golden_values.py and its workflow step do not exist on dev at all — both arrive
via main's bb5647a9b (#5989), inside the non-exempt linting job. Because the step only validates
golden files that differ from the base branch, this sync is what causes 9 dev files to be checked
for the first time. That makes the failure sync-caused, not pre-existing.

All 9 failed on non-finite iteration-time values. Those nans are placeholders for unlogged steps,
not data: model_config.yaml sets --log-interval: 2, so odd steps are never logged and step 1 has
no preceding step to time. The fix drops the nan-valued steps and re-derives the header
(start_step / end_step / step_interval), asserting the surviving keys form a single arithmetic
run. Net diff: 5 insertions / 38 deletions.

This strengthens the test rather than loosening it. common.py:239-260 compares iteration-time
by median over a steady window, mapping any non-numeric to np.inf; with nans in-window the golden
median was itself inf, so the assertion degenerated to inf == inf — a vacuous pass. Removing them
makes the comparison real over finite medians. python3 tools/check_golden_values.py now reports
"Checked 27 golden-value file(s); all values are finite."

Verification performed locally

Reproduced CI's exact lint gate (tools/autoformat.sh with CHECK_ONLY=true, which selects
megatron/core + tests/ — 415 files) using CI's pinned versions (black==24.4.2, isort==5.13.2,
pylint==3.2.6, ruff~=0.9.0):

Check	Result
black --skip-magic-trailing-comma --skip-string-normalization --check	0 (15 files reformatted; 6 arrived dirty from main, 3 are main-only new files)
isort --check	0 (3 files fixed — all pure duplicate-import removal; AST-verified no imported symbol changed)
pylint	0 (3 unused-imports fixed; pylint exits 4 on warnings and autoformat.sh runs under set -e, so these would have failed the job)
ruff check --no-fix	0
tools/check_golden_values.py	0 — 27 files, all finite
API-mismatch audit (AST, caller vs callee signatures)	0 sync-caused
Override-module reference resolution	0 unresolved
Conflict markers / unmerged paths	none

Note on pylint: param_and_grad_buffer.py's copy_tensor_to_quantized_param import was correctly
dropped — main's 8e57bb642 (#6094) deliberately replaced the per-param call with the batched
copy_tensors_to_quantized_params. fine_grained_callables.py's PreProcessNode/PostProcessNode
were kept (suppressed instead) because they are load-bearing re-exports:
tests/unit_tests/pipeline_parallel/test_pp_mhc_compatibility.py:27 imports them from that path.

API mismatch audit

Ran an AST caller/callee signature audit over all 471 changed Python files (35 findings), then the
identical audit against pristine origin/dev (31), leaving 5 merged-only candidates. All 5
reproduce on pristine origin/main at byte-identical call sites, so they are inherited from main,
not created by the merge. Two were checked further:

• moe/router.py:964 is a resolver false positive — the real callee is
  topk_routing_with_score_function (moe_utils.py:777), whose signature accepts all 10 "rejected"
  kwargs; my resolver had matched the unrelated Router.routing methods by bare name.
• models/common/utils.py is main-only yet differs from main in the merged tree because git
  rename-matched it to dev's model_chunk_schedule_plan.py and preserved dev's mHC hooks
  (preprocess_for_layer_schedule / postprocess_for_layer_schedule) plus dev's deepepv2 backend
  tuple — the desired combine-both-sides outcome.

The one real mismatch this audit caught was self-inflicted: an earlier import consolidation had
moved get_batch_on_this_tp_rank in pretrain_gpt.py into the megatron.core.utils block, assuming
both paths re-export one object. They don't — two distinct same-named functions exist with
incompatible signatures (megatron/core/utils.py:2177 takes 12+ positional params;
megatron/training/utils/common_utils.py:580 takes (data_iterator, mtp_on_this_rank=False, needs_padding_mask=False)). The call site at pretrain_gpt.py:171 uses the training one; the import
was moved back. (get_batch_on_this_cp_rank is a genuine re-export, so its differing path is safe.)

Conflict resolutions (--remerge-diff)

git 2.54.0 is installed, so --remerge-diff works. git show --remerge-diff 579e0eac7
produces 18,309 lines across 102 files — far past GitHub's PR-body limit, so the
per-file conflict census is below and reviewers can reproduce the full diff with:

git fetch origin main2dev/13_08_2026 && git show --remerge-diff 579e0eac7

Conflict resolutions by file — 40 files with the most remerge hunks (of 102)

File	remerge hunks
uv.lock	122
megatron/training/training.py	112
megatron/training/checkpointing.py	66
megatron/core/ssm/gated_delta_net/common.py	29
megatron/rl/rl_utils.py	17
tests/unit_tests/ssm/test_gated_delta_net.py	17
megatron/training/arguments.py	16
megatron/core/transformer/transformer_config.py	15
megatron/core/inference/engines/dynamic_engine.py	13
megatron/training/models/dist_utils.py	13
megatron/core/inference/contexts/dynamic_context.py	10
tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json	10
tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/golden_values_dev_dgx_gb200.json	10
tests/unit_tests/dist_checkpointing/test_optimizer.py	10
megatron/core/distributed/param_and_grad_buffer.py	8
megatron/core/transformer/moe/experts.py	8
tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json	8
tests/unit_tests/fusions/test_swiglu_fusion.py	8
tests/unit_tests/ssm/test_hybrid_layer_allocation.py	8
tests/unit_tests/transformer/moe/test_grouped_mlp.py	8
megatron/core/models/hybrid/hybrid_block.py	7
megatron/core/optimizer/optimizer.py	7
megatron/core/transformer/transformer_layer.py	7
megatron/training/argument_utils.py	7
pyproject.toml	7
tests/unit_tests/transformer/moe/test_token_dispatcher.py	7
megatron/core/fusions/fused_bias_swiglu.py	6
megatron/core/models/common/model_chunk_schedule_plan.py	6
megatron/core/transformer/multi_token_prediction.py	6
tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_h100.json	6
tests/unit_tests/transformer/moe/test_shared_experts.py	6
megatron/core/fp8_utils.py	5
megatron/core/models/common/utils.py	5
megatron/core/transformer/moe/token_dispatcher.py	5
tests/unit_tests/inference/contexts/test_dynamic_prefix_caching.py	5
tests/unit_tests/transformer/test_full_cuda_graph.py	5
docker/Dockerfile.ci.dev	4
megatron/core/models/gpt/fine_grained_callables.py	4
megatron/core/models/hybrid/hybrid_layer_allocation.py	4
megatron/core/ssm/gated_delta_net/gdn2.py	4

Full remerge diff is 18,309 lines across 102 files; the 40 files with the most conflict hunks are listed above.

uv.lock (122) and docker/Dockerfile.ci.dev (4) appear here only because their conflicts
were resolved back to dev's version; both verify byte-identical to origin/dev.
training.py (112) is the largest code entry, as expected for an override-list file.

---

Phase 3: CI fixes

get_hybrid_data_context_parallel_groups → get_dynamic_data_context_parallel_groups

The Installation Test / Pip - Python3.12 job failed with:

ImportError: cannot import name 'get_hybrid_data_context_parallel_groups' from
'megatron.core.parallel_state'. Did you mean: 'get_dynamic_data_context_parallel_groups'?

This is the function-level counterpart of the args.hybrid_context_parallel rename the skill
already mandates for training.py. Dev renamed both the accessor and its backing global
(_HYBRID_DP_CP_GROUPS → _DYNAMIC_DP_CP_GROUPS); main kept the hybrid_* spelling. Taking main's
training.py per the override list therefore left a dangling import.

Fixed at both sites, matching dev's pre-merge version verbatim (origin/dev:training.py:95 and
:2720): the import at :99 and the hybrid_cp_group_func= argument at :2445. Alphabetical import
order is preserved, so isort stays clean.

Worth noting how this surfaced: the failure is in megatron.training.__init__ → training.py, which
tests/unit_tests/conftest.py also imports — so this single dangling name would have cascaded into
every unit-test suite, not just the install check.

I then wrote a static resolver for the whole class of bug (AST, no torch needed, since the
Installation Test imports every module under megatron.*):

• Every from megatron.core.parallel_state import X across 1,601 files, plus mpu.X /
  parallel_state.X attribute reads, resolved against the 167 names parallel_state.py actually
  defines → 0 missing after the fix.
• Every intra-repo from megatron/tests/tools... import X resolved against its target module's
  top-level namespace (following re-exports) → 4 findings, all static-analysis false positives,
  identical on origin/dev and origin/main: bert_layer_with_transformer_engine_spec is served by a
  module-level __getattr__ deprecation shim (bert_layer_specs.py:95), and
  tools/bert_embedding/external_libs.py populates h5py/transformers via globals()[lib] = importlib.import_module(lib).

Two declarations the merge duplicated

build-docs (exempt from the ship gate, but a real merge artifact) reported two
autodoc2.dup_item warnings, and Sphinx runs with warnings-as-errors:

WARNING: Duplicate item core.packed_seq_params.PackedSeqParams.pad_between_seqs (attribute)
WARNING: Duplicate item core.transformer.moe.fused_a2a.HYBRIDEP_TOKEN_ALIGNMENT (data)

Both are the merge keeping dev's and main's copy of the same declaration — one occurrence on each
parent, two in the merged tree.

HYBRIDEP_TOKEN_ALIGNMENT was a harmless duplicate (identical value and comment). pad_between_seqs
was not inert: main's trailing pad_between_seqs: bool = None came after dev's
pad_between_seqs: Optional[bool] = None, so main's annotation won. The tri-state matters — the
codebase distinguishes None (infer) from False (explicitly no padding) from True:
extensions/transformer_engine.py:2264 tests packed_seq_kwargs.get("pad_between_seqs") is False,
transformer_layer.py:580-581 tests is True and is None, and test_thd_cuda_graph.py asserts
is False / is True at 10 sites.

Kept dev's Optional[bool] and dropped main's duplicate. Verified the dataclass still carries all 14
of dev's fields and all 13 of main's — nothing lost, only the duplicate removed.

To confirm this was the whole extent of the pattern, I wrote a duplicate-declaration audit (AST;
module- and class-level def/class/Assign/AnnAssign) that reports a name only when its
occurrence count in the merged tree exceeds both parents' — so pre-existing duplicates don't
create noise. Across all 474 changed Python files: exactly these 2, now 0.

Re-verification after the Phase 3 fixes

Replayed CI's real gate — tools/autoformat.sh's own file selection
(git diff --diff-filter=d --merge-base origin/main megatron/core tests/, 222 files) with CI's argv:

Check	Result
black --skip-magic-trailing-comma --skip-string-normalization --check	222 files unchanged
isort --check-only	exit 0
pylint (.pylintrc: disable=all + C0115/C0116/W0611/C0301/E0606/W0141)	10.00/10
ruff check --no-fix	All checks passed
tools/check_golden_values.py	27 files, all finite
parallel_state name resolution	0 missing
duplicate-declaration audit	0

DCO

The DCO check reports failure, but its own summary says the verdict is unknown rather than
negative:

The DCO check could not be evaluated because the complete pull request commit list could not be
retrieved. GitHub returned 250 commits from the REST API, so the app attempted to retrieve the full
list through GraphQL before evaluating the pull request. That fallback did not complete, so the DCO
verdict is unknown. … Please retry by re-running the check or pushing a new commit.

This is a pagination limit, not a missing sign-off: the PR carries 275 commits. I re-requested the
check 4 times (identical result each time) and have now pushed a new commit, which is the other remedy
the app suggests. The Phase 3 fix commit carries a Signed-off-by trailer.

For the record, the sign-off state of the commit list itself: 19 of the 274 commits inherited from
main lack a Signed-off-by trailer (e.g. 15c83d2fc "chore: rotate oncall schedule", the
Update copy-pr-bot.yaml [skip ci] commits, 648bc011f, 411a5d8b2). Those are already on main
and are not something a sync PR can alter. Prior sync PRs show the app tolerates this: #5314 passed
DCO while carrying unsigned commits from 15 different authors and two unsigned merge commits — so
merge commits and NVIDIA-member commits are not what it objects to.

-S (GPG) signing is unavailable on this runner (gpg: signing failed: No secret key), so the fix
commit is signed off with -s only — which is what DCO verifies.

[svcnvidia-nemo-ci]

/ok to test 579e0ea

[svcnvidia-nemo-ci]

/ok to test e128caa

[svcnvidia-nemo-ci]

/ok to test 6d20ae6

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.