cp: backport Nemotron 3 Ultra PRs into r0.5.0

[cuichenx]

Summary

Backports the Nemotron Ultra stack requested for r0.5.0 as one draft PR.

Included cherry-picks:

• feat(conversion): support distributed adapter export #4221: feat(conversion): support distributed adapter export
• fix(checkpointing): include optimizer scaffold while loading #4222: fix(checkpointing): include optimizer scaffold while loading
• feat(scripts): add Ultra script prerequisites #4223: feat(scripts): add Ultra script prerequisites
• feat(nemotronh): add Nemotron 3 Ultra recipes and examples #4224: feat(nemotronh): add Nemotron 3 Ultra recipes and examples

Excluded after correction:

• [docs] feat: document min GPU count for combined dense + MoE parallelism #3509
• docs: announce NVIDIA Nemotron 3 Ultra public release #4159

Validation

• git diff --check origin/r0.5.0...HEAD
• bash -n for the Ultra example shell scripts
• uv run --no-sync python -m py_compile for touched Python scripts, recipe, and tests
• pre-commit run --all-files

Notes:

• uv run pre-commit run --all-files could not resolve the release branch env on this host because nvidia-resiliency-ext==0.6.0 has no compatible wheel for this platform; direct pre-commit passed.
• Focused pytest did not run because the no-sync env lacks torch during unit-test conftest import.
• No HSG/cw-dfw validation was attempted because those clusters are unavailable per Rowan.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[cuichenx]

/ok to test 461fff5

[claude]

Light Code Review

Critical Bug

PackedPerExpertLinear.sharded_state_dict missing required pg_collection kwarg (src/megatron/bridge/peft/utils.py:2261-2263)

_make_grouped_expert_sharded_tensor now requires pg_collection as a keyword argument (it has no default value), but PackedPerExpertLinear.sharded_state_dict() does not pass it. This will raise TypeError at runtime when checkpointing any SharedOuterGroupedExpertAdapter. The fix is to either pass pg_collection=None (falls back to size-1 defaults) or thread the real collection from the owning adapter. See inline comment for a minimal fix suggestion.

Missing Test Coverage

• SharedOuterGroupedExpertAdapter and PackedPerExpertLinear are entirely new classes with no unit tests. At minimum, sharded_state_dict and forward paths for both fc1 (shared-in, per-expert-out) and fc2 (per-expert-in, shared-out) variants should be covered, especially since the sharded_state_dict bug above would have been caught.
• _make_cross_ep_replicated has no direct test. Its broadcast + grad-hook behavior is safety-critical for shared-outer adapters staying in sync across EP ranks.

Suggested test cases

No perf tests impacted.