Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/models/deepseek/deepseek-v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,14 @@ config = deepseek_v3_pretrain_config(
)
```

To place the MTP layer in a standalone PP/VPP stage instead of colocating it with loss, rebuild the recipe layout:

```python
from megatron.bridge.recipes.deepseek import set_deepseek_v3_pipeline_model_parallel_layout

set_deepseek_v3_pipeline_model_parallel_layout(config.model, mtp_standalone=True)
```

### Finetuning Recipes

Finetuning recipes for DeepSeek V3 are not currently available.
Expand All @@ -149,4 +157,3 @@ Finetuning recipes for DeepSeek V3 are not currently available.
- Recipe usage: [Recipe usage](../../recipe-usage.md)
- Customizing the training recipe configuration: [Configuration overview](../../training/config-container-overview.md)
- Training entry points: [Entry points](../../training/entry-points.md)

25 changes: 19 additions & 6 deletions docs/training/multi-token-prediction.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,16 +94,29 @@ Follow the [DCLM Tutorial](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/m

## MTP with Pipeline Parallelism

When using Pipeline Parallelism (PP), **MTP layers must be placed in the last pipeline stage** alongside the loss computation layer. Configure this using custom pipeline layout settings (`pipeline_model_parallel_split_rank`).
By default, Bridge recipes place MTP layers in the last pipeline stage alongside the loss computation layer. Recent Megatron Core versions also support placing all MTP layers in a standalone stage before the final loss stage by setting `pipeline_model_parallel_layout` with `m` entries.

### Pipeline Layout Guidelines

MTP layers take approximately the same training time as a regular transformer layer. When configuring your pipeline layout:

- **Place MTP in the last PP stage** (required for correct loss computation)
- **Reduce layers in other PP ranks** to balance computation time across stages
- Example: For a 21-layer model with PP=4 and `mtp_num_layers=1`, you might use splits like `[5, 6, 6, 4]` instead of `[5, 5, 5, 6]` to account for MTP overhead in the last stage
- Use `m` in `pipeline_model_parallel_layout` for MTP layers.
- Keep all MTP layers in the same PP/VPP stage.
- Place MTP after decoder layers and before the final loss stage.
- For standalone MTP, place an `m` stage on a non-final pipeline rank and put `L` in the final stage, for example `"E|t*3|(t|)*3tt|m|L"`.
- For colocated MTP, keep `m` with `L` in the last stage, for example `"E|t*3|(t|)*5mL"`.

DeepSeek-V3 recipes include a helper for the common layouts:

```python
from megatron.bridge.recipes.deepseek import (
deepseek_v3_pretrain_config,
set_deepseek_v3_pipeline_model_parallel_layout,
)

cfg = deepseek_v3_pretrain_config()
set_deepseek_v3_pipeline_model_parallel_layout(cfg.model, mtp_standalone=True)
```

## Parallelism Support

Expand All @@ -112,7 +125,7 @@ MTP is compatible with all major parallelism strategies in Megatron-Bridge:
| Parallelism Type | Support Status | Notes |
|------------------|----------------|-------|
| **Tensor Parallelism (TP)** | ✅ Fully Supported | MTP layers are automatically sharded across TP ranks |
| **Pipeline Parallelism (PP)** | ✅ Supported with Constraint | MTP must be in last pipeline stage (see above) |
| **Pipeline Parallelism (PP)** | ✅ Supported with Layout Constraint | MTP is colocated with loss by default, or can use a standalone `m` stage through `pipeline_model_parallel_layout` |
| **Expert Parallelism (EP)** | ✅ Fully Supported | Works with MoE models (DeepSeek-V3, Mixtral, etc.) |
| **Context Parallelism (CP)** | ✅ Fully Supported | MTP supports long-context training via CP |
| **Data Parallelism (DP)** | ✅ Fully Supported | Standard data parallelism works transparently |
Expand Down Expand Up @@ -217,7 +230,7 @@ Training instability. Try:

### Expected Log: `MTP layers not found on this PP rank`

This is normal. Only the last pipeline stage builds MTP layers.
This is normal. Only the rank selected by the pipeline layout builds MTP layers. With the default layout this is the last pipeline stage; with standalone MTP it is the stage containing `m`.

## Additional Resources

Expand Down
4 changes: 2 additions & 2 deletions examples/models/nemotron/nemotron_3/nano/conversion.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,12 @@ uv run torchrun --nproc_per_node=8 examples/conversion/convert_checkpoints_multi
--hf-model $HF_MODEL_ID \
--megatron-path ${WORKSPACE}/models/$MODEL_NAME/iter_0000000 \
--hf-path ${WORKSPACE}/models/$MODEL_NAME-hf-export \
--tp 1 --ep 8
--tp 1 --ep 8


# Round-trip validation
uv run torchrun --nproc_per_node=8 \
examples/conversion/hf_megatron_roundtrip_multi_gpu.py \
--hf-model-id $HF_MODEL_ID \
--megatron-load-path ${WORKSPACE}/models/$MODEL_NAME/iter_0000000 \
--tp 1 --ep 8
--tp 1 --ep 8
2 changes: 2 additions & 0 deletions src/megatron/bridge/recipes/deepseek/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from .deepseek_v3 import (
deepseek_v3_pretrain_config,
deepseek_v3_pretrain_config_32nodes,
set_deepseek_v3_pipeline_model_parallel_layout,
)


Expand All @@ -38,4 +39,5 @@
# DeepSeek V3
"deepseek_v3_pretrain_config",
"deepseek_v3_pretrain_config_32nodes",
"set_deepseek_v3_pipeline_model_parallel_layout",
]
50 changes: 44 additions & 6 deletions src/megatron/bridge/recipes/deepseek/deepseek_v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from typing import List, Optional, Union

import torch

from megatron.bridge import AutoBridge
Expand All @@ -26,10 +24,43 @@
from megatron.bridge.training.mixed_precision import MixedPrecisionConfig


def _build_standalone_mtp_layout(num_decoder_layers: int, total_stages: int, mtp_layers: int) -> list[list[str]]:
if mtp_layers <= 0:
raise ValueError("standalone MTP layout requires mtp_num_layers > 0")
if total_stages < 3:
raise ValueError("standalone MTP layout requires at least three PP/VPP stages")

decoder_stage_count = total_stages - 2
decoder_layers_per_stage, extra_decoder_layers = divmod(num_decoder_layers, decoder_stage_count)

layout = []
for stage_idx in range(decoder_stage_count):
num_layers = decoder_layers_per_stage + (1 if stage_idx < extra_decoder_layers else 0)
stage = ["decoder"] * num_layers
if stage_idx == 0:
stage = ["embedding"] + stage
layout.append(stage)

layout.append(["mtp"] * mtp_layers)
layout.append(["loss"])
return layout


def set_deepseek_v3_pipeline_model_parallel_layout(
model_cfg: GPTModelProvider, layout: Optional[Union[str, List[List[str]]]] = None
model_cfg: GPTModelProvider, layout: str | list[list[str]] | None = None, *, mtp_standalone: bool = False
) -> None:
"""Set the DeepSeek-V3 pipeline model parallel layout."""
"""Set the DeepSeek-V3 pipeline model parallel layout.

Args:
model_cfg: DeepSeek-V3 model configuration to update.
layout: Explicit pipeline layout. When provided, this overrides the predefined layouts.
mtp_standalone: Place MTP layers in a standalone penultimate PP/VPP stage and loss in the
final stage. Defaults to colocating MTP with loss, matching existing recipes.
"""
if layout is not None:
model_cfg.pipeline_model_parallel_layout = layout
return

mtp_layers = getattr(model_cfg, "mtp_num_layers", 1) or 0
last_layer = ["mtp"] * mtp_layers + ["loss"]
pp_size = model_cfg.pipeline_model_parallel_size or 1
Expand All @@ -43,8 +74,15 @@ def set_deepseek_v3_pipeline_model_parallel_layout(
(8, 2): [["embedding"] + ["decoder"] * 4] + [["decoder"] * 4] * 14 + [["decoder"] + last_layer],
(4, 4): [["embedding"] + ["decoder"] * 4] + [["decoder"] * 4] * 14 + [["decoder"] + last_layer],
}
if layout is not None:
model_cfg.pipeline_model_parallel_layout = layout
if mtp_standalone:
num_decoder_layers = getattr(model_cfg, "num_layers", None)
if not isinstance(num_decoder_layers, int) or isinstance(num_decoder_layers, bool) or num_decoder_layers <= 0:
raise ValueError("standalone MTP layout requires model config num_layers to be a positive integer")
model_cfg.pipeline_model_parallel_layout = _build_standalone_mtp_layout(
num_decoder_layers=num_decoder_layers,
total_stages=pp_size * vp_size,
mtp_layers=mtp_layers,
)
elif (pp_size, vp_size) in layout_map:
model_cfg.pipeline_model_parallel_layout = layout_map[(pp_size, vp_size)]

Expand Down
101 changes: 96 additions & 5 deletions tests/unit_tests/recipes/test_deepseek_recipes.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,26 @@
from typing import Callable

import pytest
from megatron.core.transformer.enums import LayerType
from megatron.core.transformer.pipeline_parallel_layer_layout import PipelineParallelLayerLayout

from megatron.bridge.recipes.deepseek import set_deepseek_v3_pipeline_model_parallel_layout
from megatron.bridge.recipes.deepseek.deepseek_v3 import _build_standalone_mtp_layout


_deepseek_module = importlib.import_module("megatron.bridge.recipes.deepseek")
_DEEPSEEK_RECIPE_FUNCS = [
getattr(_deepseek_module, name)
for name in getattr(_deepseek_module, "__all__", [])
if callable(getattr(_deepseek_module, name, None))
]
_DEEPSEEK_RECIPE_NAMES = frozenset(
{
"deepseek_v2_pretrain_config",
"deepseek_v2_lite_pretrain_config",
"deepseek_v3_pretrain_config",
"deepseek_v3_pretrain_config_32nodes",
}
)
_DEEPSEEK_EXPORTED_NAMES = set(getattr(_deepseek_module, "__all__", ()))
assert _DEEPSEEK_RECIPE_NAMES <= _DEEPSEEK_EXPORTED_NAMES
_DEEPSEEK_RECIPE_FUNCS = [getattr(_deepseek_module, name) for name in sorted(_DEEPSEEK_RECIPE_NAMES)]
assert all(callable(recipe_func) for recipe_func in _DEEPSEEK_RECIPE_FUNCS)


class _FakeModelCfg:
Expand Down Expand Up @@ -100,3 +112,82 @@ def test_each_deepseek_recipe_builds_config(recipe_func: Callable, monkeypatch:
# Parallelism and shaping
assert getattr(cfg.model, "tensor_model_parallel_size", 1) >= 1
assert getattr(cfg.model, "pipeline_model_parallel_size", 1) >= 1


def test_deepseek_v3_pipeline_layout_can_place_mtp_in_standalone_stage():
model_cfg = _FakeModelCfg()
model_cfg.num_layers = 61
model_cfg.mtp_num_layers = 1
model_cfg.pipeline_model_parallel_size = 8
model_cfg.virtual_pipeline_model_parallel_size = 2

set_deepseek_v3_pipeline_model_parallel_layout(model_cfg, mtp_standalone=True)

layout = model_cfg.pipeline_model_parallel_layout
assert layout[-2] == ["mtp"]
assert layout[-1] == ["loss"]
assert sum(stage.count("decoder") for stage in layout) == model_cfg.num_layers

parsed_layout = PipelineParallelLayerLayout(layout, pipeline_model_parallel_size=8)
assert parsed_layout.validate_layer_layout(model_cfg.num_layers, model_cfg.mtp_num_layers)
assert parsed_layout.layout[6][1] == [LayerType.mtp]
assert parsed_layout.layout[7][1] == [LayerType.loss]


def test_build_standalone_mtp_layout_rejects_too_few_total_stages():
with pytest.raises(ValueError, match="at least three"):
_build_standalone_mtp_layout(num_decoder_layers=61, total_stages=2, mtp_layers=1)


def test_build_standalone_mtp_layout_rejects_zero_mtp_layers():
with pytest.raises(ValueError, match="mtp_num_layers > 0"):
_build_standalone_mtp_layout(num_decoder_layers=61, total_stages=4, mtp_layers=0)


def test_deepseek_v3_pipeline_layout_can_place_multiple_mtp_layers_in_standalone_stage():
model_cfg = _FakeModelCfg()
model_cfg.num_layers = 61
model_cfg.mtp_num_layers = 2
model_cfg.pipeline_model_parallel_size = 4
model_cfg.virtual_pipeline_model_parallel_size = None

set_deepseek_v3_pipeline_model_parallel_layout(model_cfg, mtp_standalone=True)

layout = model_cfg.pipeline_model_parallel_layout
assert layout[-2] == ["mtp", "mtp"]
assert layout[-1] == ["loss"]
assert sum(stage.count("decoder") for stage in layout) == model_cfg.num_layers

parsed_layout = PipelineParallelLayerLayout(layout, pipeline_model_parallel_size=4)
assert parsed_layout.validate_layer_layout(model_cfg.num_layers, model_cfg.mtp_num_layers)


def test_deepseek_v3_pipeline_layout_prefers_explicit_layout_over_standalone_mtp():
model_cfg = _FakeModelCfg()
model_cfg.mtp_num_layers = 0
explicit_layout = [["embedding", "decoder", "loss"]]

set_deepseek_v3_pipeline_model_parallel_layout(model_cfg, explicit_layout, mtp_standalone=True)

assert model_cfg.pipeline_model_parallel_layout is explicit_layout


def test_deepseek_v3_pipeline_layout_requires_num_layers_for_standalone_mtp():
model_cfg = _FakeModelCfg()
model_cfg.mtp_num_layers = 1
model_cfg.pipeline_model_parallel_size = 4
model_cfg.virtual_pipeline_model_parallel_size = None

with pytest.raises(ValueError, match="num_layers"):
set_deepseek_v3_pipeline_model_parallel_layout(model_cfg, mtp_standalone=True)


def test_deepseek_v3_pipeline_layout_keeps_default_mtp_with_loss():
model_cfg = _FakeModelCfg()
model_cfg.mtp_num_layers = 1
model_cfg.pipeline_model_parallel_size = 8
model_cfg.virtual_pipeline_model_parallel_size = 2

set_deepseek_v3_pipeline_model_parallel_layout(model_cfg)

assert model_cfg.pipeline_model_parallel_layout[-1][-2:] == ["mtp", "loss"]
Loading