Skip to content

[Bugfix] glm4v_moe bridge: import GPTModelProvider instead of removed Qwen3MoEModelProvider #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CalvinXKY merged 1 commit into from
May 31, 2026
Merged

[Bugfix] glm4v_moe bridge: import GPTModelProvider instead of removed Qwen3MoEModelProvider #92

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions slime_plugins/megatron_bridge/glm4v_moe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from megatron.bridge.models.conversion.mapping_registry import MegatronMappingRegistry
from megatron.bridge.models.conversion.model_bridge import MegatronModelBridge
from megatron.bridge.models.conversion.param_mapping import AutoMapping, GatedMLPMapping, QKVMapping, ReplicatedMapping
from megatron.bridge.models.qwen.qwen_provider import Qwen3MoEModelProvider
from megatron.bridge.models.gpt_provider import GPTModelProvider
from megatron.bridge.utils.common_utils import hook_hf_module_setattr_for_tp_grad_sync
from megatron.core import parallel_state, tensor_parallel
from megatron.core.models.gpt import GPTModel as MCoreGPTModel
Expand Down Expand Up @@ -475,10 +475,10 @@ def forward(
# Model Provider (dataclass that doubles as TransformerConfig)
# ---------------------------------------------------------------------------
@dataclass
class Glm4vMoeVLModelProvider(Qwen3MoEModelProvider):
class Glm4vMoeVLModelProvider(GPTModelProvider):
"""Provider that creates Glm4vMoeVLModel.

Inherits from Qwen3MoEModelProvider to reuse MoE + TransformerConfig infra.
Inherits from GPTModelProvider to reuse MoE + TransformerConfig infra.
Defined at module level (not inside a function) so that the class is
picklable -- megatron-bridge broadcasts config objects across PP ranks
via ``torch.distributed.broadcast_object_list`` which requires pickling.
Expand Down Expand Up @@ -579,7 +579,11 @@ def provider_bridge(self, hf_pretrained):
kv_channels=getattr(text_config, "head_dim", 128),
init_method_std=text_config.initializer_range,
layernorm_epsilon=text_config.rms_norm_eps,
normalization="RMSNorm",
gated_linear_unit=True,
add_bias_linear=False,
hidden_dropout=0.0,

@gemini-code-assist gemini-code-assist Bot May 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Instead of hardcoding hidden_dropout=0.0, it is better to retrieve it from the Hugging Face text_config using getattr(text_config, "hidden_dropout", 0.0). This ensures that any custom dropout settings specified in the model configuration are respected during training/fine-tuning.

Suggested change
hidden_dropout=0.0,
hidden_dropout=getattr(text_config, "hidden_dropout", 0.0),

autocast_dtype=model_dtype,
make_vocab_size_divisible_by=self.make_vocab_size_divisible_by(text_config.vocab_size),
rotary_base=rotary_base,
rotary_percent=partial_rotary_factor,
Expand All @@ -596,6 +600,8 @@ def provider_bridge(self, hf_pretrained):
moe_shared_expert_intermediate_size=shared_expert_intermediate,
moe_layer_freq=moe_layer_freq_list,
moe_grouped_gemm=True,
moe_token_dispatcher_type="alltoall",
moe_permute_fusion=True,
moe_router_load_balancing_type="seq_aux_loss",
moe_aux_loss_coeff=0,
moe_router_score_function="sigmoid",
Expand Down
Loading