perf(recipe): align Nemotron 3 Nano library recipes

[cuichenx]

What does this PR do?

Align the public Nemotron 3 Nano library recipes with their validated H100 and
GB200 execution contracts while preserving convergence-sensitive training
semantics.

This current-main reconstruction:

• adds a shared hardware-neutral Nemotron 3 Nano pretrain/SFT/PEFT builder;
• keeps explicit H100 pretrain, SFT, and PEFT wrappers;
• retains merged feat(recipes): add Qwen3.5 and Nemotron 3 Nano GB200 pretraining #4854's GB200 pretraining implementation and adds GB200 SFT
  and PEFT wrappers plus package/top-level discovery exports;
• adds approved-field parity, intentional-divergence, denied-field, topology,
  and discovery tests;
• corrects the reduced functional fixture to TP1 when sequence parallelism is
  disabled.

The library recipes do not clone benchmark recipes wholesale. Optimizer,
schedule, routing, loss, data/sample, initialization, and BF16 contracts remain
library-owned. Only evidenced execution settings are aligned.

Recipe mapping

Library recipe	Validated execution contract
H100 pretrain, 8 GPU	TP8/PP1/CP1/EP8/ETP1, SP on, HybridEP, selective moe,layernorm recompute, eager execution, TP overlap off, unfused native vocab-parallel CE, post-step cache release for checkpoint-resume headroom.
H100 SFT / PEFT, 8 GPU	SFT TP4/SP on; PEFT TP1/SP off. DeepEP and eager packed-sequence execution.
GB200 pretrain, 8 GPU	Retains #4854's TP1/PP1/CP1/EP8/ETP1, SP-off, HybridEP, TE-scoped graph-enabled 4K implementation.
GB200 SFT / PEFT, 8 GPU	TP1/SP off, correctness-first all-to-all dispatcher, eager packed-sequence execution.

Generic Nemotron 3.5 recipes introduced on current main remain independent
of these specialized Nemotron 3 wrappers. In particular, generic Nemotron 3.5
SFT retains TP1/SP off, while its dedicated packed H100 recipe applies its own
topology.

The exact name nemotron_3_nano_pretrain_8gpu_gb200_bf16_config is exported by
both library and performance packages. Unified runner lookup follows current
main policy and selects the benchmark definition; the validated library
release workload remains uniquely available as
nemotron_3_nano_gb200_pretrain_config.

Convergence-sensitive exclusions

• No forced/debug router load balancing or equivalent biased routing.
• No changes to top-k/scaling/group policy, auxiliary-loss coefficients,
  expert-bias behavior, token dropping, capacity, optimizer, LR schedule,
  initialization, dropout, loss scaling/objective, or training samples.
• No FP8/NVFP4 conversion of the BF16 library workloads.
• No perf-only optimizer/checkpoint-memory changes. The stale overlap was
  removed because merged fix(ckpt): reduce distributed optimizer resume memory #4936 supersedes it.
• No changes inside the Megatron-LM submodule; the final tree uses current
  main's pinned revision.

Validation

Final head: c410b864f53676613872e669de3ce9cbd75c4ed5, reconstructed on
main@81c47141f9507eb0a96069f302d54f11a148fa60 with signed/DCO,
ancestry-preserving merge commits. The original public head remains an
ancestor; publication was a normal fast-forward with no rebase, force-push, or
branch rename.

Mandatory native hardware gates

Public entry point	Evidence	Result
H100 pretrain	DFW segmented chain 14300432--14300436, recovered from the preserved step-60 checkpoint by 14367322--14367325	Passed 100 optimizer steps. All 40 recovered steps 61--100 are finite with zero skipped/NaN; exact restart loads at 60/72/84/96; final loss 0.09788162, grad norm 0.261; complete step-100 checkpoint.
H100 SFT	DFW 14300430	Passed 100 steps, checkpoint, and validation; finite with zero skipped/NaN; COMPLETED 0:0.
H100 PEFT	DFW 14300431	Passed 100 steps, checkpoint, and validation; finite with zero skipped/NaN; COMPLETED 0:0.
GB200 pretrain	Merged #4854 exact current-code/MCore physical evidence	Passed 48,000 steps / 100.663296B tokens with zero skipped/NaN, exceeding the 100-step acceptance floor.
GB200 SFT	Lyris 2476997, valid segment 1	Passed 100 steps, checkpoint, and validation; finite with zero skipped/NaN; COMPLETED 0:0.
GB200 PEFT	Lyris 2476998, valid segment 1	Passed 100 steps, checkpoint, and validation; finite with zero skipped/NaN; COMPLETED 0:0.

The H100 pretraining step-72 checkpoint write initially hit a transient
filesystem/PyTorch unexpected pos failure after finite optimizer work. The
authoritative pointer remained at complete step 60, the partial checkpoint was
preserved for evidence, and the minimal recovery chain loaded exact boundaries
and completed through step 100 without recurrence.

Current-main and local verification

• Focused post-review recipe/loader selection: 163 passed.
• Broad public/performance factory, recipe, environment, and metadata matrix:
  905 passed.
• Final PR-plus-latest-main packing selection: 208 passed.
• Repeated pre-commit run --all-files: all hooks passed without changes.
• git diff --check, unmerged-entry, conflict-marker, exact-tree, DCO, and
  signature checks passed.
• Independent final review found two issues—duplicate lookup precedence and
  generic Nemotron 3.5 SFT topology inheritance. Both are fixed and covered;
  the exact final index had no remaining actionable findings.

Scope relative to merged work

• feat(recipes): add Qwen3.5 and Nemotron 3 Nano GB200 pretraining #4854 overlaps GB200 pretraining only. This PR retains that implementation
  and supplies the still-missing H100 alignment, GB200 SFT/PEFT, neutral
  builder, and parity/denied-field coverage.
• fix(ckpt): reduce distributed optimizer resume memory #4936 supersedes the stale optimizer/checkpoint-memory portion of the old PR;
  those changes are not present here.

Before this PR is ready for review

• Read and followed the contributor guidelines.
• Reconstructed the relevant work on current main and resolved conflicts.
• Added focused parity, denied-field, topology, and discovery tests.
• Completed all six mandatory native hardware gates.
• Obtained an independent final review and addressed every finding.
• Passed targeted tests and all-files pre-commit.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[cuichenx]

/ok to test c410b86