-
Notifications
You must be signed in to change notification settings - Fork 1.6k
[docs] update megatron docs #9350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -203,7 +203,8 @@ For guidance on selecting parallelization strategies, please refer to the [Train | |||||||||
| - moe_enable_deepep: Enable DeepEP for efficient token dispatching and combine in MoE models. Only works with flex token dispatcher by setting `--moe_token_dispatcher_type flex`. | ||||||||||
| - 🔥moe_grouped_gemm: When each rank contains multiple experts, multiple local GEMM kernels can be launched in parallel streams to improve utilization and performance by using GroupedLinear from TransformerEngine. Default is True. | ||||||||||
| - 🔥moe_permute_fusion: Fuses token permutation operations during token dispatch. Default is False. | ||||||||||
| - 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. **Generally, a higher value leads to worse training performance but more balanced MoE expert utilization.** Please choose an appropriate value based on experimental results. | ||||||||||
| - 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. Generally, a higher value leads to worse training performance but more balanced MoE expert utilization. Please choose an appropriate value based on experimental results. | ||||||||||
| - Note: For `moe_aux_loss`, when `padding_free` is set to False, versions of `megatron-core < 0.16` have an issue where routing loss is computed on padding tokens. Please refer to [this PR](https://github.com/NVIDIA/Megatron-LM/pull/2142). Additionally, please use `mcore-bridge >= 1.4.0.dev`. Please refer to [this PR](https://github.com/modelscope/mcore-bridge/pull/79). | ||||||||||
|
Comment on lines
+206
to
+207
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The addition of the Consider moving this information to the
Suggested change
|
||||||||||
| - moe_z_loss_coeff: Scaling coefficient for z-loss. Default is None. | ||||||||||
| - 🔥moe_shared_expert_overlap: Enables overlap between shared expert computation and the dispatcher. If not enabled, shared expert computation will be performed after routing experts. Only effective when `moe_shared_expert_intermediate_size` is set. Default is False. | ||||||||||
| - 🔥moe_expert_capacity_factor: Capacity factor for each expert. `None` means no tokens will be dropped. Default is `None`. When `--moe_expert_capacity_factor` is set, tokens exceeding an expert’s capacity will be dropped based on their selection probability. This can **balance the training load and improve training speed** (for example, set it to 1. or 2.). | ||||||||||
|
|
||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The note regarding
mcore-bridge>=1.4.0.devand the link to the context parallel (CP) support PR (PR 79) is misplaced under themoe_aux_loss_coeffparameter. Based on the code changes inmegatron_args.py, this requirement is specifically formlp_padding_freewhen used with CP.Additionally, please use backticks (
`) instead of double quotes for version numbers and package names to maintain consistency with the rest of the documentation.Consider moving the bridge version requirement to the
mlp_padding_freesection (around line 66) and updating the "Temporarily incompatible with CP" statement there.