Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/source/Megatron-SWIFT/Command-line-parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,8 @@
- moe_enable_deepep: 启用 DeepEP 以实现 MoE 模型中的高效 token 调度和合并。仅在通过设置 `--moe_token_dispatcher_type flex` 使用弹性 token 调度器时有效。
- 🔥moe_grouped_gemm: 当每个rank包含多个专家时,通过在多个流中启动多个本地 GEMM 内核,利用 TransformerEngine中的GroupedLinear提高利用率和性能。默认为True。
- 🔥moe_permute_fusion: 在令牌分发过程中融合令牌重排操作。默认为False。
- 🔥moe_aux_loss_coeff: 默认为0,不使用aux_loss。**通常情况下,该值设置的越大,训练效果越差,但MoE负载越均衡**,请根据实验效果,选择合适的值。
- 🔥moe_aux_loss_coeff: 默认为0,不使用aux_loss。通常情况下,该值设置的越大,训练效果越差,但MoE负载越均衡,请根据实验效果,选择合适的值。
- 注意:moe_aux_loss在`padding_free`为 False 的情况下,"megatron-core<0.16"存在对padding token计算routing loss的情况,参考[这个PR](https://github.com/NVIDIA/Megatron-LM/pull/2142)。此外请使用"mcore-bridge>=1.4.0.dev",参考[这个PR](https://github.com/modelscope/mcore-bridge/pull/79)。
Comment on lines +195 to +196

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The note regarding mcore-bridge>=1.4.0.dev and the link to the context parallel (CP) support PR (PR 79) is misplaced under the moe_aux_loss_coeff parameter. Based on the code changes in megatron_args.py, this requirement is specifically for mlp_padding_free when used with CP.

Additionally, please use backticks (`) instead of double quotes for version numbers and package names to maintain consistency with the rest of the documentation.

Consider moving the bridge version requirement to the mlp_padding_free section (around line 66) and updating the "Temporarily incompatible with CP" statement there.

Suggested change
- 🔥moe_aux_loss_coeff: 默认为0,不使用aux_loss。通常情况下,该值设置的越大,训练效果越差,但MoE负载越均衡,请根据实验效果,选择合适的值。
- 注意:moe_aux_loss在`padding_free`为 False 的情况下,"megatron-core<0.16"存在对padding token计算routing loss的情况,参考[这个PR](https://github.com/NVIDIA/Megatron-LM/pull/2142)。此外请使用"mcore-bridge>=1.4.0.dev",参考[这个PR](https://github.com/modelscope/mcore-bridge/pull/79)
- 🔥moe_aux_loss_coeff: 默认为0,不使用aux_loss。通常情况下,该值设置的越大,训练效果越差,但MoE负载越均衡,请根据实验效果,选择合适的值。
- 注意:moe_aux_loss在`padding_free`为 False 的情况下,`megatron-core < 0.16`存在对padding token计算routing loss的情况,参考[这个PR](https://github.com/NVIDIA/Megatron-LM/pull/2142)

- moe_z_loss_coeff: z-loss 的缩放系数。默认为None。
- 🔥moe_shared_expert_overlap: 启用共享专家计算与调度器通信之间的重叠。如果不启用此选项,共享专家将在路由专家之后执行。仅在设置了`moe_shared_expert_intermediate_size`时有效。默认为False。
- 🔥moe_expert_capacity_factor: 每个专家的容量因子,None表示不会丢弃任何token。默认为None。通过设置 `--moe_expert_capacity_factor`,超出专家容量的 token 会基于其被选中的概率被丢弃。可以**令训练负载均匀,提升训练速度**(例如设置为1或2)。
Expand Down
3 changes: 2 additions & 1 deletion docs/source_en/Megatron-SWIFT/Command-line-parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,8 @@ For guidance on selecting parallelization strategies, please refer to the [Train
- moe_enable_deepep: Enable DeepEP for efficient token dispatching and combine in MoE models. Only works with flex token dispatcher by setting `--moe_token_dispatcher_type flex`.
- 🔥moe_grouped_gemm: When each rank contains multiple experts, multiple local GEMM kernels can be launched in parallel streams to improve utilization and performance by using GroupedLinear from TransformerEngine. Default is True.
- 🔥moe_permute_fusion: Fuses token permutation operations during token dispatch. Default is False.
- 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. **Generally, a higher value leads to worse training performance but more balanced MoE expert utilization.** Please choose an appropriate value based on experimental results.
- 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. Generally, a higher value leads to worse training performance but more balanced MoE expert utilization. Please choose an appropriate value based on experimental results.
- Note: For `moe_aux_loss`, when `padding_free` is set to False, versions of `megatron-core < 0.16` have an issue where routing loss is computed on padding tokens. Please refer to [this PR](https://github.com/NVIDIA/Megatron-LM/pull/2142). Additionally, please use `mcore-bridge >= 1.4.0.dev`. Please refer to [this PR](https://github.com/modelscope/mcore-bridge/pull/79).
Comment on lines +206 to +207

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The addition of the mcore-bridge>=1.4.0.dev requirement and the link to the CP support PR (PR 79) under moe_aux_loss_coeff is confusing. This version requirement is related to mlp_padding_free support for context parallelism, not the auxiliary loss coefficient.

Consider moving this information to the mlp_padding_free section (around line 69) and updating the note about CP incompatibility there.

Suggested change
- 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. Generally, a higher value leads to worse training performance but more balanced MoE expert utilization. Please choose an appropriate value based on experimental results.
- Note: For `moe_aux_loss`, when `padding_free` is set to False, versions of `megatron-core < 0.16` have an issue where routing loss is computed on padding tokens. Please refer to [this PR](https://github.com/NVIDIA/Megatron-LM/pull/2142). Additionally, please use `mcore-bridge >= 1.4.0.dev`. Please refer to [this PR](https://github.com/modelscope/mcore-bridge/pull/79).
- 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. Generally, a higher value leads to worse training performance but more balanced MoE expert utilization. Please choose an appropriate value based on experimental results.
- Note: For `moe_aux_loss`, when `padding_free` is set to False, versions of `megatron-core < 0.16` have an issue where routing loss is computed on padding tokens. Please refer to [this PR](https://github.com/NVIDIA/Megatron-LM/pull/2142).

- moe_z_loss_coeff: Scaling coefficient for z-loss. Default is None.
- 🔥moe_shared_expert_overlap: Enables overlap between shared expert computation and the dispatcher. If not enabled, shared expert computation will be performed after routing experts. Only effective when `moe_shared_expert_intermediate_size` is set. Default is False.
- 🔥moe_expert_capacity_factor: Capacity factor for each expert. `None` means no tokens will be dropped. Default is `None`. When `--moe_expert_capacity_factor` is set, tokens exceeding an expert’s capacity will be dropped based on their selection probability. This can **balance the training load and improve training speed** (for example, set it to 1. or 2.).
Expand Down
4 changes: 3 additions & 1 deletion swift/megatron/arguments/megatron_args.py
Original file line number Diff line number Diff line change
Expand Up @@ -623,7 +623,9 @@ def _set_default(self):
'mcore-bridge>=1.3.0.dev',
'Please install mcore-bridge>=1.3.0.dev to use mlp_padding_free with sequence parallel.')
if self.context_parallel_size > 1:
raise ValueError('mlp_padding_free is not compatible with context parallel.')
require_version(
'mcore-bridge>=1.4.0.dev',
'Please install mcore-bridge>=1.4.0.dev to use mlp_padding_free with context parallel.')
if self.local_rank is None:
self.local_rank = get_dist_setting()[1]
if self.lr is None:
Expand Down
Loading