Migrate Qwen3 dense and MoE models to HybridModel

[Phlip79]

Summary

• add shared Qwen HybridModel layer-layout and pipeline-balancing helpers
• migrate Qwen3 dense and Qwen3 MoE bridges from GPTModel to HybridModel
• map each logical decoder block to separate attention and MLP/MoE physical layers
• preserve logical layer counts during Hugging Face export and retain MTP mappings
• update focused bridge, helper, and recipe tests

Models modified

Dense Qwen3 checkpoints using Qwen3ForCausalLM:

• Qwen/Qwen3-0.6B
• Qwen/Qwen3-1.7B
• Qwen/Qwen3-4B
• Qwen/Qwen3-8B
• Qwen/Qwen3-14B
• Qwen/Qwen3-32B

Qwen3 MoE checkpoints using Qwen3MoeForCausalLM:

• Qwen/Qwen3-30B-A3B
• Qwen/Qwen3-235B-A22B

Stack

This is PR 1 of 4. Merge the stack in order:

1. Migrate Qwen3 dense and MoE models to HybridModel #4747 - Qwen3 dense and MoE
2. Migrate Qwen3-Next and Qwen3.5 models to HybridModel #4836 - Qwen3-Next and Qwen3.5 text
3. Migrate Qwen VL models to HybridModel #4837 - Qwen3-VL and Qwen3.5-VL
4. Migrate Qwen3-ASR and Qwen3-Omni to HybridModel #4838 - Qwen3-ASR and Qwen3-Omni

Validation

• added full-test-suite and needs-more-tests labels
• https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/57855069

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Phlip79]

/ok to test

[copy-pr-bot]

/ok to test

@Phlip79, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[Phlip79]

/ok to test cf3a16b

[Phlip79]

/ok to test 8af13ca

[Phlip79]

/ok to test a47b094

[Phlip79]

/ok to test e46c136

[claude]

qwen_moe_layer_symbols has no caller anywhere in src/ or tests/ — both qwen3_moe_bridge.py and qwen3_bridge.py pass a scalar mlp_symbols=Symbols.MOE/Symbols.MLP, so the decoder_sparse_step / mlp_only_layers dense-in-MoE handling here is untested dead code. Either wire it into the MoE bridge (Qwen MoE configs do carry decoder_sparse_step and mlp_only_layers, and a checkpoint with dense layers would currently be mapped as all-MoE and fail conversion) or drop it and add a direct unit test if it's intended for a follow-up.

[claude]

Review: Qwen3 / Qwen3-MoE migrated to HybridModel

Light review focused on correctness, tests, and doc drift. The core change is sound: Qwen3 dense + MoE bridges now target HybridModel with a two-physical-layer-per-logical-block layout, PP segmentation keeps each logical block on one stage, and the MTP-combine is now correctly deferred until MTP is actually enabled. Explicit per-layer mappings and the new test_qwen_hybrid.py give good coverage.

Findings

• [Low] Dead / untested helper: qwen_hybrid.py:163 qwen_moe_layer_symbols (decoder_sparse_step / mlp_only_layers) has no caller in src/ or tests/; both bridges pass a scalar mlp_symbols. A real Qwen-MoE checkpoint with dense (mlp_only_layers) layers would currently be mapped as all-MoE and fail conversion. Wire it into the MoE bridge or drop it (see inline comment).

• [Question] Doc drift: docs/fern/versions/nightly/pages/bridge-tech-details.mdx (~216-232) still shows the Qwen3 bridge registered against target=GPTModel and returning a GPTModelProvider. Per the CONTRIBUTING refactor checklist, docs referencing the old target/provider should be refreshed. (The 0.4.2 versioned docs are frozen and can be left.)

Notes / verified

• mtp_use_repeated_layer is a valid MCore TransformerConfig field (used by nemotronh recipes), so the new finalize() guard is safe.
• hf_config is a real class attribute on MegatronModelBridge (set during dispatch); the autouse fixtures patching Bridge.hf_config are consistent with that.
• Functional-test override guarding (hasattr before setattr) is satisfied via run_pretrain_recipe_test; hybrid_layer_pattern is a real provider field.
• FLOPs calculator already handles hybrid_layer_pattern via get_hybrid_layer_counts.
• Recipes build the provider from AutoBridge.from_hf_pretrained(...), so no recipe edits needed.

Suggested test cases

• test_qwen3_4b_ckpt_mcore
• test_qwen_pretrain_recipes[qwen3_600m_config-qwen3_600m]
• test_seqpacking_cp_example
• test_modelopt_spec_preserves_grouped_moe_topology
• TestMegatronQwen3Bridge::test_provider_bridge_basic
• TestMegatronQwen3Bridge::test_provider_bridge_with_custom_kwargs
• TestMegatronQwen3Bridge::test_mapping_registry_has_qwen3_specific_mappings
• TestMegatronQwen3Bridge::test_qwen3_qk_norm_mapping_difference
• TestMegatronQwen3Bridge::test_qwen3_no_qkv_bias_mapping
• TestQwen3BridgeMTPMapping
• TestQwen3MoEBridge::test_provider_bridge_basic
• TestQwen3MoEBridge::test_provider_bridge_235b_config
• TestQwen3MoEBridge::test_mapping_registry_parameter_mappings
• TestQwen3MoEBridge::test_mapping_registry_moe_mappings
• test_mtp_pattern_is_deferred_when_mtp_is_disabled
• test_mtp_pattern_honors_recipe_override_set_after_conversion
• test_pipeline_segmentation_preserves_logical_blocks_and_embedding_loss_balance
• test_pipeline_segmentation_rejects_stale_explicit_segments

No scripts/performance/configs/ files are touched: no perf tests impacted.

[Phlip79]

/ok to test 83caabb