Skip to content

[docs] feat: document min GPU count for combined dense + MoE parallelism - #3509

Merged
cuichenx merged 1 commit into
mainfrom
chcui/parallelism-skill-min-gpus
Apr 27, 2026
Merged

[docs] feat: document min GPU count for combined dense + MoE parallelism#3509
cuichenx merged 1 commit into
mainfrom
chcui/parallelism-skill-min-gpus

Conversation

@cuichenx

Copy link
Copy Markdown
Contributor

Summary

Updates skills/perf-techniques/parallelism-strategies/SKILL.md with a new Minimum GPU Count section.

The dense path uses a TP*CP-mesh and the MoE path uses an EP*ETP-mesh; within each PP stage these two meshes share the same GPUs rather than multiplying. The correct minimum is therefore:

```
min_gpus = PP * max(TP * CP, EP * ETP)
```

not the common (and incorrect) PP * TP * CP * EP * ETP that shows up in READMEs and slurm sizing tables.

Changes

  • New Minimum GPU Count section with the formula, an explicit "wrong" callout, and a worked example table covering common dense + EP combos.
  • Extends the existing implicit DP formula to include expert_data_parallel_size = world_size / (PP * EP * ETP) for the MoE path.
  • Adds a "Scaling above the minimum" subsection showing how adding GPUs scales DP / EDP.
  • Cites Pangu Ultra MoE (arXiv:2504.14960) where the decoupling of attention and MoE meshes is described.
  • Adds an entry to the existing Pitfalls list pointing readers at the new section.

Test plan

  • Docs-only change; no runtime impact.
  • Pre-commit passes (markdown is not linted; whitespace / EOF hooks skip).

Adds a "Minimum GPU Count" section clarifying that the dense (TP*CP) and
MoE (EP*ETP) meshes share the same GPUs within each PP stage, so:

    min_gpus = PP * max(TP * CP, EP * ETP)

not the naive product PP * TP * CP * EP * ETP that frequently appears
in READMEs and slurm sizing. Also extends the implicit DP formula to
include EDP for the MoE path, adds an example table, explains how
adding GPUs scales DP / EDP above the minimum, and references Pangu
Ultra MoE (arXiv:2504.14960) where the decoupling is described.

Signed-off-by: Chen Cui <chcui@nvidia.com>
@yaoyu-33 yaoyu-33 added docs-only With great power comes great responsibility. needs-review PR is ready for code review and waiting on a reviewer labels Apr 27, 2026
@cuichenx
cuichenx merged commit 15fa4bd into main Apr 27, 2026
82 checks passed
@cuichenx
cuichenx deleted the chcui/parallelism-skill-min-gpus branch April 27, 2026 20:05
vasunvidia pushed a commit to vasunvidia/Megatron-Bridge that referenced this pull request Jun 10, 2026
…ism (NVIDIA-NeMo#3509)

Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-only With great power comes great responsibility. needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants