Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/api-guide/core/generalized_tensor_parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ GTP_remat runs under both the standard **Adam** `DistributedOptimizer` and **Muo

- **Adam** shards optimizer state over the gtp_remat/egtp_remat-excluded replicate group, like any GTP_remat run (§3.2).
- **Muon** keeps matrix params *whole* (Newton–Schulz needs the full 2D weight). A GTP_remat-replicated whole param (e.g. MoE router, latent-proj MLPs by default) then lands on one checkpoint key shared by all GTP_remat peers, so the LayerWise optimizer folds `gtp_rank` into its `replica_id` — exactly one peer writes (the optimizer-state analog of the model-side fold in §3.3).
- **Native-FP8 optimizer-state matching (Muon path).** The save-side dequantize (§3.3) hands DCP a *fresh* BF16 tensor, which breaks the id-based optimizer-param → model-`ShardedTensor` match for every native-FP8 GTP_remat weight. The dequantized copy carries a `_gtp_dequant_src` backlink to the live FP8 param, and `_backfill_gtp_sharded_param_map` reuses the model's **own** entry (backlink first, tagged-name second) — preserving its full offsets (expert axes included) and `replica_id`. Only truly-unmatched params (Mamba `in_proj`, a gathered+split factory) take the per-shard rebuild, which refuses expert-parallel params rather than emit EP-colliding shards.
- **Native-FP8 optimizer-state matching (Muon path).** The save-side dequantize (§3.3) hands DCP a *fresh* BF16 tensor, which breaks the id-based optimizer-param → model-`ShardedTensor` match for every native-FP8 GTP_remat weight. The dequantized copy carries a `_gtp_dequant_src` backlink to the live FP8 param, and `_backfill_gtp_sharded_param_map` reuses the model's **own** entry (backlink first, tagged-name second) — preserving its full offsets (expert axes included) and `replica_id`. Only truly-unmatched params (the SSM `in_proj` weights, gathered+split factories) take the per-shard rebuild, which refuses expert-parallel params rather than emit EP-colliding shards.

Neither path adds a GTP_remat-specific checkpoint format or call site.

Expand Down Expand Up @@ -543,7 +543,9 @@ Because the offsets reconstruct the global shape, the checkpoint is independent

**Alignment padding & cross-topology reshard.** When `_gtp_slice_one_param` pads `out_features` to a multiple of `gtp_remat_size · pad_for_alignment`, the saved global describes the *padded* shape, so the helper sets `allow_shape_mismatch=True`. DCP then tolerates a load-side topology whose alignment yields a different padded size — the unpadded data overlaps and the tail pad rows are zeros GTP_remat recomputes.

> Note: Mamba's `in_proj` is a special case: it **all-gathers its GTP_remat shards** back to the logical TP-local size and strips the pad *before* saving, so its global is topology-independent and needs no `allow_shape_mismatch`.
> Note: the SSM `in_proj` weights — Mamba's (`mamba_mixer.py`, split `[z|x|B|C|dt]`) and gated-delta-product's (`gated_delta_product.py`, split householder-major into `z|V*|K*|Q|b*|a`) — are a special case: each **all-gathers its GTP_remat shards** back to the logical TP-local size and strips the pad *before* saving, so its global is topology-independent and needs no `allow_shape_mismatch`. This is required, not just tidier: the split-chunk boundaries do not line up with the GTP_remat slice boundaries, so a raw shard cannot be split at all. The checkpoint therefore matches a non-GTP_remat run byte-for-byte.
>
> On **load**, the split factory's `merge_fn` is wrapped to invert this: it cats the chunks back to the unpadded TP-local width, re-pads with zeros up to `gtp_remat_local_size · gtp_remat_size`, and slices by the GTP_remat rank — mirroring `_gtp_slice_one_param` so the tensor lands in the live shard's layout. `gtp_remat_size == 1` skips both the gather and the pad/slice.

**Optimizer state.** The distributed optimizer's master/moment `ShardedObject`s are keyed by `dp_group_idx`. Under GTP_remat/EGTP_remat each peer owns a *different* master shard (the optimizer shards over the gtp_remat/egtp_remat-**excluded** replicate group), so the index is taken from the gtp_remat/egtp_remat-**merged** model-parallel group (`mp_group` for dense, `expt_tp_pp_with_egtp_remat_group` for expert) — giving every peer a distinct key while replicate-group ranks remain true replicas under that key.

Expand Down Expand Up @@ -739,7 +741,7 @@ torchrun --nproc-per-node 4 -m pytest tests/unit_tests/generalized_tensor_parall
| `test_gtp_grad_correctness.py` | Gradient + dist-opt + grad-norm numeric parity vs a DP baseline at replicate (DP) > 1. Also the fp32-accumulation reduce-scatter (§2.6): gtp_remat-axis and DDP-axis parity, plus the size-2 bypass. |
| `test_gtp_cudagraph_grad.py` | Capture-step grad-norm guard (§1.2): `_backup_grads_before_capture`/`_restore_grads_after_capture` keep a graph capture from clobbering finalized `main_grad` (own params + cross-graph `next_w`, incl. routed-expert `weight_list`). |
| `test_gtp_partial_cg.py` | Four-layer partial-CG loss and eager-vs-replay grad-norm parity with two-slot ring reuse across independently replayed graphs (§3.5). |
| `test_gtp_dcp.py` | DCP sharding metadata (§3.3): TP×GTP_remat offsets, pad reshard, `replica_id`, native-FP8 save/load. |
| `test_gtp_dcp.py` | DCP sharding metadata (§3.3): TP×GTP_remat offsets, pad reshard, `replica_id`, native-FP8 save/load. Also the SSM `in_proj` gather+split: the gated-delta-product mixer's factory build/merge at MXFP8 alignment, and a full DCP save→load roundtrip of that mixer. |
| `test_gtp_muon_dcp.py` | Muon optimizer-state DCP roundtrip (§1.6): `replica_id` fold + native-FP8 backfill matching. |
| `test_gtp_recompute_chain.py` | Recompute-chain buffers (§3.1): adjacent nodes never share a gather buffer, dense and grouped, plus dgrad/wgrad parity vs no-recompute. |
| `test_gtp_mtp.py` | GTP_remat + MTP shared weights (§3.5), 14 cases over `mtp_use_repeated_layer` × dense/MoE. Both MTP hazards are silent, so each needs its own guard: the async reduce-scatter path is compared numerically against the sync path on an identical model/sharding/batch, and all-gathers issued are tallied against consumes to catch a consume reading a buffer nothing gathered into. |
Expand Down
85 changes: 84 additions & 1 deletion megatron/core/ssm/gated_delta_product.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,25 @@
get_cu_seqlens,
)
from megatron.core.tensor_parallel import get_cuda_rng_tracker
from megatron.core.tensor_parallel.gtp_api import HAVE_GTP
from megatron.core.transformer import TransformerConfig
from megatron.core.transformer.module import MegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
from megatron.core.transformer.utils import (
ensure_metadata_has_dp_cp_group,
make_sharded_tensors_for_checkpoint,
sharded_state_dict_default,
)
from megatron.core.utils import deprecate_inference_params, is_using_quantization_scales
from megatron.core.utils import (
deprecate_inference_params,
is_using_quantization_scales,
make_tp_sharded_tensor_for_checkpoint,
)

if HAVE_GTP:
from megatron.core.tensor_parallel.gtp_api import is_gtp_param
else:
is_gtp_param = None

try:
from causal_conv1d import causal_conv1d_fn, causal_conv1d_update
Expand Down Expand Up @@ -932,6 +943,9 @@ def _get_states_from_cache(self, inference_context, batch_size, *, inference_par

def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
"""Provide a sharded state dictionary for distributed checkpointing."""
# Guard for cases metadata is not provided
metadata = ensure_metadata_has_dp_cp_group(metadata)

sharded_state_dict = {}
# Parameters
self._save_to_state_dict(sharded_state_dict, "", keep_vars=True)
Expand Down Expand Up @@ -968,6 +982,41 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
+ (1 + self.num_householder) * self.ngroups_local_tp * self.d_state
+ self.nheads_local_tp * (1 + self.num_householder)
)
# Under GTP, in_proj.weight is GTP-sliced along axis 0. The [z|V*|K*|Q|b*|a] split
# boundaries don't line up with GTP slice boundaries, so gather the shards back to
# TP-local size (strip the trailing pad rows from the gathered tail) and fall through
# to the same split path the non-GTP run uses — saved ckpt matches a non-GTP run.
in_proj_gtp_remat_size = getattr(self.in_proj.weight, "gtp_remat_size", 1)
in_proj_is_gtp = (
in_proj_gtp_remat_size > 1 and HAVE_GTP and is_gtp_param(self.in_proj.weight)
)
if in_proj_is_gtp:
gtp_remat_group = self.in_proj.weight.group
# in_proj.weight was already built at the sharded size by the submodule
# sharded_state_dict above — and, for native-FP8 GTP, dequantized to BF16 there
# (make_tp_sharded_tensor_for_checkpoint). Gather those (BF16) shards back to the
# full TP-local size so the [z|V*|K*|Q|b*|a] split below matches a non-GTP run.
local = sharded_state_dict[f"{prefix}in_proj.weight"].data.contiguous()
gathered = torch.empty(
(local.shape[0] * in_proj_gtp_remat_size,) + local.shape[1:],
dtype=local.dtype,
device=local.device,
)
torch.distributed.all_gather_into_tensor(gathered, local, group=gtp_remat_group)
if gathered.shape[0] != in_proj_dim:
gathered = gathered[:in_proj_dim].contiguous()
# Gathered weight is replicated across full dp_cp; replica_id needs only the DP slot.
dp_cp_rank = torch.distributed.get_rank(metadata["dp_cp_group"])
sharded_state_dict[f"{prefix}in_proj.weight"] = make_tp_sharded_tensor_for_checkpoint(
gathered,
f"{prefix}in_proj.weight",
tp_axis=0,
replica_id=(0, 0, dp_cp_rank),
prepend_offsets=sharded_offsets,
tp_group=self.pg_collection.tp,
dp_cp_group=metadata["dp_cp_group"],
)

assert sharded_state_dict[f"{prefix}in_proj.weight"].data.size(0) == in_proj_dim, (
in_proj_dim,
sharded_state_dict[f"{prefix}in_proj.weight"],
Expand Down Expand Up @@ -999,6 +1048,40 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
sharded_state_dict[key], in_proj_split_sections, in_proj_split_names, 0
)

# GTP load-side inverse of the save-time all-gather (see
# docs/api-guide/core/generalized_tensor_parallel.md §3.3, in_proj note): the checkpoint
# stores the FULL TP-local in_proj.weight (pad stripped) under the per-householder split
# keys, so the default merge_fn cats them back to ``in_proj_dim`` rows with no
# padding. To reload into the live GTP param we must mirror init
# (``_gtp_slice_one_param``): F.pad the merged tensor with zeros up to
# ``gtp_remat_local_size * gtp_remat_size``, then slice by ``gtp_remat_local_rank``.
# gtp_remat_size=1 has no pad/slice.
if in_proj_is_gtp:
factory = sharded_state_dict[f"{prefix}in_proj.weight"]
gtp_remat_local_rank = torch.distributed.get_rank(self.in_proj.weight.group)
gtp_remat_local_size = self.in_proj.weight.data.size(0)
original_merge_fn = factory.merge_fn

@torch.no_grad()
def _gtp_slice_after_cat(
sub_state_dict,
_orig=original_merge_fn,
_rank=gtp_remat_local_rank,
_size=gtp_remat_local_size,
_gtp_remat_size=in_proj_gtp_remat_size,
):
full = _orig(sub_state_dict)
aligned_total = _size * _gtp_remat_size
pad_rows = aligned_total - full.shape[0]
if pad_rows > 0:
full = torch.nn.functional.pad(full, (0, 0, 0, pad_rows))
start = _rank * _size
return full[start : start + _size].contiguous()

sharded_state_dict[f"{prefix}in_proj.weight"] = replace(
factory, merge_fn=_gtp_slice_after_cat
)

conv_dim = (
self.d_inner_local_tp * self.num_householder
+ (1 + self.num_householder) * self.ngroups_local_tp * self.d_state
Expand Down
Loading
Loading