feat(recipes): add Qwen3.5 and Nemotron 3 Nano GB200 pretraining

[cuichenx]

Summary

• add library Qwen3.5 text pretraining recipes for dense 9B and 35B-A3B on 8x GB200
• add the validated Nemotron 3 Nano BF16 pretraining recipe for 8x GB200
• export stable recipe aliases for direct users and NeMo-CI selection
• preserve library recipe behavior; no perf_recipes or performance-harness files are added

Configuration

Qwen3.5-9B uses Qwen/Qwen3.5-9B-Base, TP1/PP1/CP1/EP1, GBS128/MBS2, sequence length 4,096, native fused cross entropy, BF16 gradient reduction, scoped Transformer Engine graphs over attention/MLP, and disabled loss/gradient NaN checks as validated by the full convergence run.

Qwen3.5-35B-A3B uses the corresponding text-only MoE provider with TP1/PP1/CP1/EP8, learned routing, HybridEP, MBS1, canonical flex-dispatcher SMS configuration, FP32 router computations, and scoped TE graphs.

Nemotron 3 Nano uses TP1/PP1/CP1/EP8, GBS512/MBS2, sequence length 4,096, HybridEP, native fused cross entropy, BF16 gradient reduction, and scoped TE graphs over attention, Mamba, router, and MoE preprocessing modules.

Validation

• all-file pre-commit passes on current main
• focused Qwen/Nano recipe tests, including the public Qwen text alias and exact 48k runtime overrides: 92 passed
• exhaustive offline recipe-factory matrix: 252 passed
• fresh independent origin/main...HEAD review; the one FP32-router finding was fixed in 84dd8aea3, followed by re-review with no remaining actionable findings
• Qwen3.5-9B physical 8x GB200 validation: 100/100 RP2 steps, LM loss 13.24286 -> 6.583383, zero skipped/NaN iterations (W&B hvdq6oa3)
• Qwen3.5-9B physical 64x GB200 convergence: 48,000/48,000 steps, 100.663296B RP2 tokens, final checkpoint saved, final LM loss 2.094276, zero skipped/NaN iterations (W&B xbp3sunp)
• Nemotron 3 Nano physical 64x GB200 convergence: 48,000/48,000 steps, 100.663296B RP2 tokens, final checkpoint saved, final LM loss 2.027169, zero skipped/NaN iterations (W&B avbr31km)
• Qwen3.5-35B-A3B physical 8x GB200 validation: 100/100 RP2 steps, LM loss 12.83251 -> 6.099396, zero skipped/NaN iterations (W&B 5w1agqzm)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[claude]

LGTM - clean, well-tested text-only Qwen3.5-35B-A3B GB200 pretraining recipe with a compatibility alias, a dedicated unit test, and docs.

What I verified:

• Every assigned model/optimizer/ddp config field is a real field used in other recipes (glm45, qwen_vl perf recipes); no phantom-field typos.
• The architectures override Qwen3_5MoeForCausalLM matches the registered causal-LM bridge in models/qwen/qwen35_bridge.py, so AutoBridge selects the text bridge rather than the VLM.
• Unified/legacy MoE SM-count handling (moe_flex_dispatcher_num_sms=32, moe_hybridep_num_sms=None) is consistent with _normalize_moe_dispatcher_sm_config in training/model_load_save.py.
• Test patching is correct: the gb200 module imports both AutoBridge and AutoConfig at top level, and the new qwen35 guard correctly exempts this recipe from the cross_entropy_fusion_impl==te assertion (it uses native).
• The 8-GPU (EP=8) recipe is not added to the 2-GPU functional smoke list, so CI will not try to download the base model.

Minor question (non-blocking):

• The text recipe uses Qwen/Qwen3.5-35B-A3B-Base while the VL recipe uses Qwen/Qwen3.5-35B-A3B (no -Base). The -Base variant is right for pretraining, but please confirm that HF repo id exists.

Note: the _FakeModelCfg/_FakeProvider fakes accept arbitrary attribute assignment, so a future typo in a model config field name would not be caught by these unit tests; a pre-existing repo pattern, not new to this PR.

Suggested test cases:

• No perf tests impacted (no files under scripts/performance/configs/ or perf base configs are touched).
• tests/unit_tests/recipes/qwen/test_qwen35_recipes.py::test_qwen35_text_recipe_uses_nested_language_model_config
• test_qwen_recipes.py::test_each_qwen_recipe_builds_config with qwen35_35b_a3b_pretrain_config
• test_qwen_recipes.py::test_each_qwen_recipe_builds_config with qwen35_35b_a3b_pretrain_8gpu_gb200_bf16_config

[cuichenx]

Added the convergence-parity fix in 80d8c4b: the Nemotron 3 Nano GB200 pretraining recipe now defers model vocabulary sizing to the runtime tokenizer, restoring the 32K RP2 tokenizer behavior instead of inheriting the 131072-entry HF checkpoint vocabulary. Validation: focused recipe regression test passed in the 26.08 RC2 container; all-file pre-commit passed.

[cuichenx]

/ok to test b723f28