[do not merge] backport Qwen3.5 and Nemotron 3 Nano pretraining to r0.5.0

[cuichenx]

What changed

Backports the Qwen3.5 text-only and Nemotron 3 Nano GB200 library pretraining recipes onto r0.5.0 for NeMo 26.06 convergence validation.

Qwen3.5

• adds Qwen3.5 9B and 35B-A3B GB200 library recipes from feat(recipes): add Qwen3.5 and Nemotron 3 Nano GB200 pretraining #4854
• exposes canonical qwen35_9b_pretrain_config and qwen35_35b_a3b_pretrain_config aliases
• adapts the provider annotation and HybridEP SM field to the older release layout

Nemotron 3 Nano

• adds nemotron_3_nano_pretrain_8gpu_gb200_bf16_config and the NeMo-CI selector alias nemotron_3_nano_gb200_pretrain_config
• fixes model and dataset sequence length at 4096
• uses the validated GB200 topology: TP1, PP1, CP1, EP8, ETP1, sequence parallelism off
• enables the release HybridEP path with moe_hybridep_num_sms=16
• keeps eager execution because scoped CUDA graphs exceed memory at the library recipe MBS2
• explicitly keeps forced router load balancing disabled
• retains the release library optimizer, LR schedule, learned routing policy and coefficient, token semantics, BF16/FP32 optimizer-state contract, GBS3072, and MBS2

NeMo-CI should select this recipe with MODEL_RECIPE_NAME=nemotron_3_nano_gb200. The shared release template then deliberately overrides only the convergence-run controls to 48,000 steps, sequence length 4096, and GBS512; the model case overrides GPU count to 64 and MBS to 2.

Why

NeMo-CI needs a reviewable release-branch SHA for the paired Qwen3.5 9B and Nemotron 3 Nano 100B-token, 64-GPU DGX Cloud GB200 convergence candidates against the 26.06 container. This does not replace the corresponding main-targeting recipe work.

Validation

• 92 focused Qwen/Nemotron recipe and discovery tests passed in nvcr.io/nvidian/nemo:26.06.01.rc2 with release-pinned Megatron-Core 458c8d0e
• public scripts/performance/run_recipe.py dry-run resolved nemotron_3_nano_gb200 with the exact 64-GPU NeMo-CI overlay: GBS512, MBS2, sequence length 4096, and 48,000 steps
• the resolved Nemotron config confirms TP1/PP1/CP1/EP8/ETP1, HybridEP, eager execution, and force balancing disabled
• all-file pre-commit and git diff --check passed
• the same GB200 Nemotron execution recipe completed the corrected 100-step public-entry hardware gate on 8 GB200 GPUs as Lyris job 2382796: 100/100, finite loss and grad norm, zero skipped/NaN iterations, checkpoint and validation/test complete
• Qwen3.5 9B previously completed 100 real-RP2 steps on 8 GB200 GPUs as Lyris job 2378268, W&B hvdq6oa3

Active convergence run

The corrected Qwen-only NeMo-CI overlay root 58144288 is pinned to Qwen head dd81e664e and is unaffected by this Nemotron addition. A Nemotron launch must use current PR head 987244934 and MODEL_RECIPE_NAME=nemotron_3_nano_gb200.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.