feat(nemotronh): add Nemotron 3 Ultra recipes and examples

[cuichenx]

Summary

• add Nemotron 3 Ultra pretraining, packed OpenMath SFT, and packed OpenMath PEFT recipes
• add Ultra docs and Slurm examples for conversion, inference, DCLM pretraining, SFT, PEFT, and packing intended to run from the main branch
• add recipe unit tests for Ultra pretrain, SFT, and PEFT starting parallelism and recompute settings

This PR is stacked on #4223 so its diff stays limited to Ultra-specific recipe/docs/examples.

Blast Radius / Test Assessment

• Touches model recipes, docs, and launch scripts for a new Ultra example workflow.
• L0/unit coverage is needed for recipe construction and is included for pretrain, SFT, and PEFT recipe defaults.
• Because the launch scripts are large multi-node workflows, full validation is manual/cluster-based rather than CI functional tests; no L1/L2 CI labels are added.
• Manual validation expected: checkpoint conversion, base inference, DCLM pretraining, packed OpenMath SFT, and packed OpenMath PEFT on target hardware.

Validation

• uv run --no-sync pre-commit run --files docs/models/nemotron/index.md docs/models/nemotron/nemotron3-ultra.md examples/models/nemotron/nemotron_3/README.md examples/models/nemotron/nemotron_3/ultra/README.md examples/models/nemotron/nemotron_3/ultra/conversion.sh examples/models/nemotron/nemotron_3/ultra/pack_data_job.sh examples/models/nemotron/nemotron_3/ultra/slurm_conversion.sh examples/models/nemotron/nemotron_3/ultra/slurm_inference.sh examples/models/nemotron/nemotron_3/ultra/slurm_peft.sh examples/models/nemotron/nemotron_3/ultra/slurm_pretrain.sh examples/models/nemotron/nemotron_3/ultra/slurm_sft.sh src/megatron/bridge/recipes/nemotronh/__init__.py src/megatron/bridge/recipes/nemotronh/nemotron_3_ultra.py tests/unit_tests/recipes/nemotronh/test_nemotron_3_ultra.py
• uv run --no-sync pre-commit run --files tests/unit_tests/recipes/nemotronh/test_nemotron_3_ultra.py
• uv run --no-sync python -m py_compile src/megatron/bridge/recipes/nemotronh/nemotron_3_ultra.py tests/unit_tests/recipes/nemotronh/test_nemotron_3_ultra.py
• bash -n examples/models/nemotron/nemotron_3/ultra/conversion.sh examples/models/nemotron/nemotron_3/ultra/pack_data_job.sh examples/models/nemotron/nemotron_3/ultra/slurm_conversion.sh examples/models/nemotron/nemotron_3/ultra/slurm_inference.sh examples/models/nemotron/nemotron_3/ultra/slurm_peft.sh examples/models/nemotron/nemotron_3/ultra/slurm_pretrain.sh examples/models/nemotron/nemotron_3/ultra/slurm_sft.sh
• grep -n "Day-0\|day-0\|MEGATRON_BRIDGE_BRANCH\|uv sync" examples/models/nemotron/nemotron_3/ultra/README.md || true confirmed the obsolete setup text is gone.

Focused pytest was attempted with uv run --no-sync python -m pytest tests/unit_tests/recipes/nemotronh/test_nemotron_3_ultra.py -q, but pytest is not installed in this local no-sync environment.

[claude]

Light Code Review -- Overall this is a clean PR. Recipes, Slurm examples, docs, and tests are internally consistent and follow the established Nemotron 3 patterns. -- Missing test coverage: nemotron_3_ultra_pretrain_config is imported, exported, and documented in the README, but has no unit test. The existing Super recipe test covers all three recipe functions; the Ultra test only covers SFT and PEFT. Adding a pretrain test would guard the MTP settings, default parallelism (TP=1, PP=3, EP=8), and optimizer hyperparameters. See inline comment on test_nemotron_3_ultra.py. -- No other issues found. Shell scripts are well-structured. Documentation accurately reflects the recipe defaults and Slurm script parameters. -- Suggested test cases: No perf tests impacted. Unit tests to add: test_pretrain_uses_initial_parallelism_values asserting TP=1, PP=3, EP=8, MTP settings, optimizer defaults for nemotron_3_ultra_pretrain_config

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.