chore: AUT-2472 nightly sync main into dev (2026-10-02)

[svcnemo-autobot]

Background

Nightly sync of the oldest exact 100-commit dev..main slice into dev, following megatron-lm:nightly-sync.

• dev base: f80139c3efc3c0a03c9669df5db5e310f68ad4c7
• slice tip (main): ab20a601247647ab8aab56af1dc0f3adb9e9193a. Commit 101, ec53920e, is excluded.

What changed

• 100 main commits are imported unchanged.
• Signed merge 6cd8c361, with parents f80139c3 (dev) and ab20a601 (main).
• One squashed, signed repair commit, be3eaaf9:
  • Code: parallel_state, dynamic_context, layer_wise_optimizer and training.py.
  • moe/fused_a2a.py: removes a duplicated HYBRIDEP_TOKEN_ALIGNMENT constant. This merge accident broke build-docs.
  • F1, CI error extraction: restores main's generate_jet_trigger_job.py and .gitlab/stages/02, 03 and 04 (main 06bb3997a, ci: Enhance nightly/mr/weekly error reporting #5831). The imported triage tooling (ci: integrate nemo-ci-triage with linear issues management for gitlab failures #5881, ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow #5957) needs them. 01.build.yml stays on dev, keeping the 26.04 pin.
  • F5, from review 5470751474: fine_grained_callables.py now releases NCCL-EP zero-copy tensors correctly. Test: test_combined_1f1b_tensor_release.py.
  • F6, from review 5470751474: optimizer/__init__.py and emerging_optimizers.py split QKV for Muon under GTP_remat, with a supporting transformer_config.py change. Tests: test_emerging_optimizers.py and test_transformer_config.py.
  • Tests: 3 more unit-test files for the merge repair.
  • Golden values: 10 files (5 dgx_h100, 5 dgx_gb200).
    • The NaN values that failed linting are fixed.
    • The 3 dgx_gb200 moe files now hold the values that come with main cd4afffa (Fix gradient-norm undercounting when using EP and TP #5916). They were taken from the CICD 37688059691 artifacts and match dev sync 5ccdc5527 (chore: AUT-2153 sync exact 100 main commits into dev #7094).
  • gb200 gpt-dynamic-inference.yaml: adds the netrc setting.

Details

Commit graph relative to dev: 102 commits = 100 imports + 1 merge + 1 repair. This was checked again on be3eaaf9.

Imported commits, oldest first (git rev-list --reverse f80139c3..ab20a601)

e443bd3 bd0872d cfb116f 5a88c57 337c061 7012adb 12c05a2 368fa88 58bf14e ddaa315 a046281 d207685 cc5b092 96d4159 890247c e4ea859 b842f59 8d16c67 602fad0 bb5647a 6cd6ea5 52cf20d 411a5d8 c5398de 51e915a 15d4b34 acd0e35 4464d1c 4f72cc0 ff60360 81770cb a69bbb0 0a7f372 4f65717 6f0db6f 72411e5 7ee2580 650b783 2d754e6 af3d240 648bc01 066145e dfe9d04 50cc27a 21fe0fe 10c9383 7e6a045 c5ff22b cd4afff 7ec1bce 543b803 076f61f 876016a fc57daa 10cd30c 4b18b26 07e958f c4b4d59 866efa4 6f7bcd4 bacd340 8a424b8 4da9212 7557c02 053fa07 541d5ee c922805 2576c15 818b258 2658704 157c023 bc67abd 8a63dd5 0a447f5 f43ff6b b78cfd5 3ff70c0 54045a5 efb3963 c7d1865 e598899 e1b2a2f 542ff53 6183f9d eab6551 3f88509 8f1c99c 77c8772 9436656 8e57bb6 354b5d2 ab16414 ff6b92a 3fd8f33 175b4cc ce8a8a7 9d416ee 0081c84 dc3c9ab ab20a60

• Reviews 5447733392 and 5454819572: F1 is fixed. F2 and F3 are byte-identical to main (77c87725 Add disaggregated KV transfer backends #5861, a69bbb09 Add encoder prefetch for heterogeneous MIMO training #5833), so they should be fixed on main. F4 (648bc011 Inference: Optimized triton kernels to extract mamba states in prefix caching #5866) is already fixed on main by f6c33bde4 (test(determinism): add kernel replay tests and a PR coverage gate #7148), which comes after this slice.
• Review 5470751474: F5 and F6 are fixed in be3eaaf9.
• Kept from dev: CODEOWNERS, pyproject.toml, uv.lock, Dockerfile.ci.dev and the dev golden values. The TE, nemo-run and emerging_optimizers pins are unchanged. multi-approval-bot.yml stays deleted.
• Taken from main: notify.py, Dockerfile.linting and the F1 files.
• Advisory checks: 2885 dev lines flagged on be3eaaf9, up from 2880 on d92c81ed. The 5 new lines are dev-only optimizer lines that F6 rewrites on purpose. There are no CODEOWNERS or dependency-pin warnings.
• Remerge-diff: 99 files, all reviewed, and the repair leaves it unchanged. merge-tree against the live dev tip f11f0340 is clean.

Tested

Local H100 runs, image sha256:4a507fcf41bd3b89cb540bed1c1d80e4df4fed0b31e058290b073a5783eb3434.

On be3eaaf9, for the unit groups F5/F6 touch: 5/5 passed.

• **/*.py: 2153 passed
• pipeline_parallel: 203 passed
• a2a_overlap: 46 passed
• models: 659 passed
• transformer: 2250 passed

Other rows were not rerun because F5/F6 does not affect them; their earlier passes are kept.

Earlier, on the merge repair:

• Lint: passed.
• Unit: 30/30 passed.
• Functional: 78/79 passed. common/moe_perf fails the same way on both merge parents (a pre-existing RandomSTE.generator AttributeError).

Not tested locally:

• The NCCL-EP zero-copy path end to end.
• A real multi-rank GTP run.
• Local lint for be3eaaf9.

GitHub CI:

• On d92c81ed: CICD 37758767237 had 158 jobs pass and 3 skipped. The only failure is cicd-mbridge-testing (the Megatron-Bridge App installation lookup failed). common/moe_perf hits the same AttributeError there; CI marks it optional.
• On be3eaaf9: pending.

[svcnvidia-nemo-ci]

Follow-up result

F1 is fixed in the inspected source. The generator now captures the workload's exit status immediately, makes extraction non-blocking, accepts the opt-in option, and conditionally publishes the expected artifacts. All four stage argument arrays enable it, matching the existing regression assertions.

The three findings below from review 5447733392 remain supported by the current head. Importing their implementations unchanged from main does not resolve them.

This completes the targeted light follow-up, not a whole-PR review. The previously reviewed tree is unavailable, so the complete inter-review delta could not be reconstructed independently. Inspection covered the prior counterexamples, identifiable subsequent repairs, and their directly affected contracts/tests. No new escalation-eligible regression was established. No tests were executed; acceptance checks and suggested code below require author verification. The required local style guide and determinism-testing integration guide are absent from the protected base tree.

F2 — Make NCCL KV source election agree between sender and receiver

Failure: Give source ranks 0 and 1 identical TP1/PP1 KV shards, and destination rank 2 one complete shard. Call each sender's begin_push_blocks with rank 2's metadata, and call the receiver's begin_pull_blocks with both source metadata records. The sender enumeration plans with only its own source layout, so both senders post sends. The receiver supplies both sources to the planner's smallest-rank election, so it receives only from rank 0. Rank 1 consequently has an unmatched send, and its handle cannot complete normally. Expected: exactly one elected source per replicated shard, with every posted send matched by a receive. This is a backend API counterexample; an integrated production-server path was not established.

Correction: Change _kv_transfers and the push metadata contract so the push side receives the complete source roster or an explicitly agreed elected-source set. Both sides must use the same source election and filter transfers for their own rank. Update metadata producers and direct callers to supply that information. The current push contract carries destination metadata only, so replacing the singleton planner argument alone is insufficient; an exact local-only patch would be misleading.

Preservation: Deduplicate source replicas, not destination replicas: every destination still needs complete KV. Retain nonreplicated TP2→TP1 head merging, explicit PP layer windows, block ordering, and TP/PP metadata normalization. Do not blanket-change the separate conv/recurrent SSM enumeration.

Acceptance — proposed checks: Using the existing test geometry and block IDs [1, 3], mock P2P posting for the replicated-source setup above. Rank 0 must post the required sends, rank 1 none, and receiver operations must match send peers, shapes, counts, and per-peer order. Reversing the source roster must elect rank 0 again. With two destination replicas, both must receive complete data. Retain the real-GPU TP2→TP1 test, specifically torch.equal(buf, expected) and the final all-rank success assertion; add a bounded-time replicated-source transfer check.

F3 — Restore exception-safe MIMO teardown

Failure: In MIMO main, cleanup follows pretrain only on normal return. A setup exception skips distributed shutdown; a training exception after the data provider starts prefetch also skips loader closure and topology destruction. A stubbed pretrain that starts the mocked data provider and raises reproduces the missing cleanup calls. The merge-base implementation guaranteed teardown with nested finally blocks.

Correction: In main, initialize topology = None alongside prefetch_loader = None, put the existing initialization-through-pretrain statements inside try, and replace the trailing cleanup with this suggested block:

finally:
    try:
        if prefetch_loader is not None:
            prefetch_loader.close()
    finally:
        try:
            if topology is not None:
                topology.destroy()
        finally:
            shutdown_distributed()

The target is lines 81–148; retain the setup, model-capture hook, nested data provider, and pretrain arguments unchanged apart from indentation. The invariant is that every applicable cleanup is attempted on success or failure, and distributed shutdown is still attempted when an earlier cleanup raises.

Preservation: Keep cleanup order—loader, topology, distributed state—and preserve the loader's bounded, idempotent close. Non-prefetch runs must not attempt loader closure, and failure before topology assignment must not attempt topology destruction.

Acceptance — proposed checks: Mock setup resources and record cleanup calls for successful training, setup failure before topology assignment, and training failure after a mocked loader starts. Assert the applicable calls occur exactly once in the specified order; the original training exception must propagate when cleanup succeeds. Inject loader-close and topology-destroy exceptions separately and assert shutdown still runs. Retain the stuck-worker regression: close warns without raising on join timeout, and producer-stream synchronization remains uncalled.

F4 — Add the required bit-exact extraction-kernel tests

Coverage gap: The two Triton extraction kernels are used by Mamba prefix-cache extraction. Their direct tests perform single reference comparisons, not repeated-execution bit-exact checks. The current determinism suite still lacks the kernel-test directory and manifest required by the pinned light rubric. This is a testing gap, not evidence that either kernel is nondeterministic.

Correction: Add repeated-execution kernel tests under the required determinism-kernel test location and register both extraction kernels in its manifest. Keep kernel implementations unchanged unless those tests expose a failure. The integration guide and registry schema are unavailable, so exact registration code cannot safely be prescribed.

Preservation and acceptance — proposed checks: For each public extraction function, use identical immutable inputs and independently sentinel-initialized outputs, synchronize, and compare output bytes exactly across repeated executions. Cover runtime counts zero, partial, and full; repeated/nonmonotonic SSM indices; and supported state dtypes, including BF16 and FP32. Capture once and replay with changed runtime counts: only active slots may change, while padded slots must retain their sentinel. Preserve the existing conv-clamp assertions for pos=2, d_conv=4, which must read token positions [0, 0, 0, 1], and both existing real-count gating assertions. Also retain the multi-chunk prefix-cache regression: the token-768 snapshot must be registered, and the reuse request must match three Mamba blocks.

Review generated by codex

[svcnemo-autobot]

Reply to review 5454819572 (F2–F4 follow-up)

Thanks — F1 is confirmed fixed. I checked F2–F4 again, this time against current main (159b443c) as well as this head. I'm leaving them unchanged in this sync PR:

Finding	Introduced on main by	This head (d92c81ed)	Current main 159b443c	Disposition
F2: NCCL KV source election	77c87725 (#5861)	nccl.py is byte-identical to slice tip ab20a601. kv_reshard.py is unchanged since the merge-base.	The push side still calls plan_kv_reshard([self._layout], sources) (L190).	Bug owned by main; fix it on main
F3: MIMO teardown	a69bbb09 (#5833) replaced the nested finally that the merge-base f41ec549 and dev shared (blob 4f304958).	pretrain_mimo.py is byte-identical to the slice tip.	Cleanup still runs in sequence with no finally. 6 later main commits touch this file.	Regression owned by main; fix it on main, then a later sync brings it to dev
F4: determinism kernel tests	648bc011 (#5866)	tests/unit_tests/determinism/kernels/ is absent on dev, on the slice and on this head.	Added by f6c33bde4 (#7148), which is after this slice. Its manifest.py registers ssm/ops/common/intermediate_extraction.py.	Already fixed on main; arrives in a later sync slice

Why not patch these here: this PR imports exactly 100 main commits, adds one merge commit, and allows at most one repair commit, limited to merge and compatibility fixes.

• A dev-only fix for F2 or F3 would make dev diverge from main in files main is still changing, so later syncs would hit conflicts.
• An F4 fix would mean importing commits beyond this slice.

The right fix is a PR on main for F2 and one for F3. If maintainers want, I can prepare them as separate work. If you agree with this disposition, please dismiss this bot review. No code changed for F2–F4, so no tests were run for them.

[FDecaYed]

Two potential integration issues I found in the consolidated merge at d92c81ed. This review is limited to merge-introduced issues, not defects already present on either baseline.

1. NCCL-EP zero-copy protection is bypassed by the active overlap schedule

The schedule plan instantiates the GPT-local TransformerLayerNode, whose super call does not forward moe_ncclep_zero_copy. The new protection therefore remains disabled on the active path.

There is a second gap: scheduled release runs before the symmetric-storage check, and its release action can attempt to resize persistent symmetric storage to zero. Main's zero-copy buffers and dev's release/ownership machinery need to be integrated consistently.

Suggested correction: Forward the flag through the active GPT node and treat persistent symmetric buffers as externally owned, excluding them from destructive scheduled release and incompatible alias ownership checks. If that combination cannot be supported safely in this sync, reject zero-copy + scheduled release explicitly. Verify overlap with scheduled release both enabled and disabled, including buffer reuse.

2. GTP-sharded QKV parameters silently lose Muon splitting

The retained QKV tagger reconstructs logical rows using TP only. For example, 8 heads, 4 query groups and head dimension 128 give 2048 logical QKV rows; TP1/GTP2 stores 1024 rows locally, so the tagger sets is_qkv=False and silently bypasses the requested QKV split.

Suggested correction: Keep GTP and manually adapt the core fix from #7117, which is outside this 100-commit slice:

• Derive eligibility and split metadata from the unsharded, unpadded logical layout.
• Gather before splitting/NS, then restore padding and the local GTP shard.
• Preserve dev #7085's per-head/MLA handling and existing mode defaults; carry required metadata through optimizer master copies.

Please avoid replacing the whole Muon implementation or reverting GTP wholesale: #6057 in this slice depends on #4967. A focused compatibility fix should be enough. Acceptance should exercise the actual optimizer factory and compare matched TP/GTP QKV updates, including padding, per-head and fragmented layouts, with consistent collectives across ranks.

These findings are based on source inspection and focused CPU-only logic checks; no GPU/NCCL end-to-end reproduction was performed. The proposed corrections and acceptance checks have not been implemented or run.

[FDecaYed]

Approving the overall sync at d92c81ed. I don't consider the two integration issues in my earlier review blockers for this PR, but ideally they should be fixed before merging:

1. NCCL-EP zero-copy buffer protection in the active overlap/scheduled-release paths.
2. GTP-aware Muon QKV tagging and gather-before-split handling.

For the second issue, #7117 is already merged into main, so its fix can also be brought in by a subsequent main-to-dev sync.

[ko3n1g]

/ok to test be3eaaf