feat(model): add MiniMax M3 language-model bridge and recipes by khazic · Pull Request #4652 · NVIDIA-NeMo/Megatron-Bridge

khazic · 2026-07-05T05:08:50Z

What does this PR do ?

Add Megatron-Bridge support for the MiniMax M3 language model (MiniMaxAI/MiniMax-M3): HF to Megatron bridge, pretraining and SFT recipes, AutoBridge registration matrix coverage, unit tests, and a docs page. First implementation step for #4585.

Changelog

Add MiniMaxM3Bridge (src/megatron/bridge/models/minimax_m3/), registered for MiniMaxM3SparseForConditionalGeneration. It converts the language_model.* weights of the multimodal checkpoint to a Megatron-Core GPTModel:
- Mixed dense/MoE decoder (first 3 layers dense via moe_layer_freq), 128 routed experts with top-4 routing, sigmoid scoring with expert-bias correction and routed_scaling_factor on the normalized top-k weights (DeepSeek-V3-style), plus one shared expert.
- SwiGLU-OAI activation mapped to activation_func=quick_gelu with activation_func_clamp_value and glu_linear_offset (same mechanism as GPT-OSS; M3 expert weights are non-interleaved and the on-disk layout is per-expert w1/w3/w2 under the legacy block_sparse_moe prefix, as in MiniMax-M2).
- Gemma-style RMSNorm (x * (1 + w)) via layernorm_zero_centered_gamma, per-head QK RMSNorm via stock qk_layernorm, partial RoPE (rotary_dim / head_dim), GQA with 4 KV heads.
- Export produces a standalone text-only checkpoint (MiniMaxM3VLForCausalLM with full_attention layer types).
Add pretrain and SFT recipes under src/megatron/bridge/recipes/minimax/ (h100 namespace, TP=2 / PP=4 / EP=32 baseline; 60 layers split evenly across 4 stages).
Add MiniMaxM3SparseForConditionalGeneration to the AutoBridge registration matrix (string registration).
Add unit tests under tests/unit_tests/models/minimax_m3/ covering provider config mapping (FFN size split, MoE routing flags, activation, norms, rotary percent, MTP disable, mlp_layer_types fallback), mapping registry contents and language_model. prefixing, and text-only export config.
Add docs/models/minimax/minimax-m3.md plus the fern mirror and nav entries.

Known limitations (documented in the model page and bridge docstring)

Language model only: the CLIP-style vision tower, multimodal projector, and patch-merge MLP are not mapped.
The lightning-indexer block-sparse attention branch (self_attn.index_*) is not mapped; the Megatron model runs full causal attention on every layer. Block selection keeps index_topk_blocks * index_block_size (2048) key tokens per query, so full attention is mathematically identical up to that sequence length and an approximation beyond it.
MTP modules are not mapped (the released checkpoint advertises num_nextn_predict_layers but ships no mtp.* weights), so mtp_num_layers is forced to None.
The MXFP8 variant is not supported; use the bf16 checkpoint.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)

Additional Information

Related to [model] Add Megatron-Bridge support for MiniMax M3 #4585
Conversion roundtrip and generation verification on a real checkpoint (GPU) is planned as a follow-up; functional/convergence tests and the remaining [model] Add Megatron-Bridge support for MiniMax M3 #4585 checklist items (vision tower, block-sparse attention kernel, MTP) are follow-up work.

Add Megatron-Bridge support for the MiniMax-M3 text backbone (MiniMaxM3SparseForConditionalGeneration checkpoints): - MiniMaxM3Bridge: HF <-> Megatron conversion for the language_model weights of the multimodal checkpoint. Covers mixed dense/MoE layers (first 3 dense), 128 routed experts (top-4, sigmoid scoring with expert-bias correction and routed scaling), one shared expert, per-head QK RMSNorm, Gemma-style zero-centered RMSNorm, partial RoPE, and the clamped SwiGLU-OAI activation (quick-GELU gate with clamp and +1 linear offset, same mechanism as GPT-OSS). - Pretrain and SFT recipes under recipes/minimax (TP=2, PP=4, EP=32 baseline for H100). - AutoBridge registration matrix coverage and unit tests. - Docs page under docs/models/minimax and the fern mirror. Known limitations (documented): vision tower, lightning-indexer block-sparse attention branch, and MTP modules are not mapped; the Megatron model runs full causal attention, which matches the sparse kernel exactly up to index_topk_blocks * index_block_size tokens. Related to NVIDIA-NeMo#4585 Signed-off-by: khazic <khazzz1c@gmail.com>

copy-pr-bot · 2026-07-05T05:08:53Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yaoyu-33 · 2026-07-05T05:34:45Z

+        load_weights=False
+    )
+
+    _apply_minimax_m3_model_settings(cfg)


dont need this extra layer of function, just flatten it here.

Done, flattened into both recipe functions in ad3eda0.

yaoyu-33 · 2026-07-05T05:35:15Z

+    return cfg
+
+
+def minimax_m3_sft_128gpu_h100_bf16_config() -> ConfigContainer:


sft better just provide thd.

Done, switched the SFT dataset to packed THD in ad3eda0.

…HD for SFT Address review feedback: inline the shared model settings into each recipe function and switch the SFT dataset to packed (THD) sequences. Signed-off-by: khazic <khazzz1c@gmail.com>

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-07-06T21:23:32Z

/ok to test 471d942

Signed-off-by: khazic <khazzz1c@gmail.com>

yaoyu-33 · 2026-07-08T17:27:39Z

/ok to test 3e4bd01

upstream/main removed src/megatron/bridge/models/hf_pretrained/vlm.py and PreTrainedVLM; VLM bridges now type their HF handle as PreTrainedCausalLM (the wrapper only reads .config here, which PreTrainedCausalLM exposes). The stale PreTrainedVLM import broke conftest collection (ModuleNotFoundError) and failed every unit-test job. Merge upstream/main and switch the import. Signed-off-by: khazic <khazzz1c@gmail.com>

yaoyu-33 · 2026-07-10T20:11:55Z

/ok to test f1fe523

…bridges The SUPPORTS_HF_PRETRAINED_EXPORT capability gate resolved self._model_bridge unconditionally at the top of save_hf_pretrained, before the config-only branch. Resolving the bridge requires a concrete, registered architecture, so a config-only save from a bare PretrainedConfig (no architectures, or an auto_map pointing at an unregistered class) raised ValueError / NotImplementedError from bridge resolution instead of writing config.json. This broke the pre-existing test_save_hf_pretrained_config_only* tests. Treat an unresolvable bridge as "nothing to gate" and fall through to the normal path; still raise NotImplementedError when a bridge resolves and reports it cannot export (e.g. the MiniMax M3 LM-only bridge), so the export-rejection contract is preserved. Signed-off-by: khazic <khazzz1c@gmail.com>

yaoyu-33 · 2026-07-13T16:41:29Z

/ok to test 9370b6a

L0_Launch_training_finetune failed only on test_lora_finetune_with_packed_sequences with a NCCL/NET-MLX5 SIGSEGV during init; unrelated to this MiniMax-M3 change. Re-running. Signed-off-by: khazic <khazzz1c@gmail.com>

…-NeMo#4652) Signed-off-by: khazic <khazzz1c@gmail.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>

github-actions Bot added the community-request label Jul 5, 2026

yaoyu-33 reviewed Jul 5, 2026

View reviewed changes

yaoyu-33 added area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Jul 5, 2026

khazic and others added 2 commits July 5, 2026 17:01

refactor(recipe): flatten MiniMax-M3 recipe settings and use packed T…

ad3eda0

…HD for SFT Address review feedback: inline the shared model settings into each recipe function and switch the SFT dataset to packed (THD) sequences. Signed-off-by: khazic <khazzz1c@gmail.com>

fix(model): validate MiniMax M3 conversion and inference

471d942

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 previously approved these changes Jul 6, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public July 6, 2026 21:24 Inactive

copy-pr-bot Bot had a problem deploying to test July 6, 2026 21:24 Error

copy-pr-bot Bot temporarily deployed to public July 6, 2026 22:10 Inactive

copy-pr-bot Bot temporarily deployed to public July 6, 2026 22:11 Inactive

copy-pr-bot Bot temporarily deployed to public July 6, 2026 22:43 Inactive

ci: add license header to test_hf_to_megatron_generate_text.py

3e4bd01

Signed-off-by: khazic <khazzz1c@gmail.com>

khazic dismissed yaoyu-33’s stale review via 3e4bd01 July 7, 2026 02:44

copy-pr-bot Bot temporarily deployed to public July 8, 2026 17:28 Inactive

copy-pr-bot Bot temporarily deployed to test July 8, 2026 17:28 Inactive

copy-pr-bot Bot temporarily deployed to public July 8, 2026 18:27 Inactive

copy-pr-bot Bot temporarily deployed to public July 8, 2026 18:28 Inactive

copy-pr-bot Bot temporarily deployed to public July 8, 2026 18:56 Inactive

khazic added 2 commits July 9, 2026 15:59

Merge remote-tracking branch 'upstream/main' into khazic/feat/minimax-m3

bf6065b

copy-pr-bot Bot temporarily deployed to public July 10, 2026 20:12 Inactive

copy-pr-bot Bot temporarily deployed to test July 10, 2026 20:13 Inactive

copy-pr-bot Bot temporarily deployed to public July 10, 2026 20:52 Inactive

copy-pr-bot Bot temporarily deployed to public July 10, 2026 21:19 Inactive

khazic added 2 commits July 13, 2026 16:46

Merge branch 'main' into khazic/feat/minimax-m3

9370b6a

yaoyu-33 added ready-to-merge PR is approved, current, and only waiting for CI to pass before merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Jul 13, 2026

copy-pr-bot Bot temporarily deployed to public July 13, 2026 16:48 Inactive

copy-pr-bot Bot temporarily deployed to test July 13, 2026 16:50 Inactive

copy-pr-bot Bot temporarily deployed to public July 13, 2026 17:17 Inactive

copy-pr-bot Bot temporarily deployed to public July 13, 2026 17:18 Inactive

copy-pr-bot Bot temporarily deployed to public July 13, 2026 17:49 Inactive

ci: re-trigger CI (unrelated LoRA packed-seq infra flake)

a70a92d

L0_Launch_training_finetune failed only on test_lora_finetune_with_packed_sequences with a NCCL/NET-MLX5 SIGSEGV during init; unrelated to this MiniMax-M3 change. Re-running. Signed-off-by: khazic <khazzz1c@gmail.com>

yaoyu-33 merged commit da3faa8 into NVIDIA-NeMo:main Jul 14, 2026
1 check passed

khazic deleted the khazic/feat/minimax-m3 branch July 14, 2026 02:13

cuichenx mentioned this pull request Jul 28, 2026

[NeMo FW 26.08 Release] MBridge 0.6.0 Roadmap #5144

Open

pruprakash mentioned this pull request Aug 22, 2026

docs(misc): list EXAONE, Kimi K3, MiniMax-M3 and Nemotron-3 Ultra as supported #5743

Open

Navigation Menu

Uh oh!

feat(model): add MiniMax M3 language-model bridge and recipes - #4652

feat(model): add MiniMax M3 language-model bridge and recipes#4652
yaoyu-33 merged 9 commits into
NVIDIA-NeMo:mainfrom
khazic:khazic/feat/minimax-m3

khazic commented Jul 5, 2026

Uh oh!

copy-pr-bot Bot commented Jul 5, 2026

Uh oh!

yaoyu-33 Jul 5, 2026

Uh oh!

khazic Jul 5, 2026

Uh oh!

yaoyu-33 Jul 5, 2026

Uh oh!

khazic Jul 5, 2026

Uh oh!

yaoyu-33 commented Jul 6, 2026

Uh oh!

yaoyu-33 commented Jul 8, 2026

Uh oh!

yaoyu-33 commented Jul 10, 2026

Uh oh!

yaoyu-33 commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return cfg


		def minimax_m3_sft_128gpu_h100_bf16_config() -> ConfigContainer:

Uh oh!

Conversation

khazic commented Jul 5, 2026

What does this PR do ?

Changelog

Known limitations (documented in the model page and bridge docstring)

GitHub Actions CI

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Jul 5, 2026

Uh oh!

yaoyu-33 Jul 5, 2026

Choose a reason for hiding this comment

Uh oh!

khazic Jul 5, 2026

Choose a reason for hiding this comment

Uh oh!

yaoyu-33 Jul 5, 2026

Choose a reason for hiding this comment

Uh oh!

khazic Jul 5, 2026

Choose a reason for hiding this comment

Uh oh!

yaoyu-33 commented Jul 6, 2026

Uh oh!

yaoyu-33 commented Jul 8, 2026

Uh oh!

yaoyu-33 commented Jul 10, 2026

Uh oh!

yaoyu-33 commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants