Extend FP32 MoE numerics to low-precision recipes and registered activations - #77

Draft

zianglih wants to merge 31 commits into

radixark:miles-mainfrom

zianglih:agent/moe-fp32-low-precision

Commits on Feb 25, 2026

[1/8] fix: misc compatibility fixes for PyTorch and TE (#2)

authored and
guapisolo
committed
b4d401c
[2/8] feat: support partial checkpoint loading (#3)

authored and
guapisolo
committed
6c3fb8c

Commits on Feb 26, 2026

[3/8] feat: add post-attention and post-MLP layernorm support (#4)

yueming-yuan
authored and
guapisolo
committed
3f0fcac
[4/8] fix: MLA RoPE triton kernel head indexing and v_dim=0 support (#5)

authored and
guapisolo
committed
e82d9ee
[5/8] feat: support MTP training in RL (#6)

authored and
guapisolo
committed
3011052
[6/8] feat: support rollout routing replay (R3) and bypass for MTP layers (#10)

guapisolo
and
yueming-yuan
committed
228a364
[7/8] feat: add INT4 fake QAT for MoE grouped linear (#9)

authored and
guapisolo
committed
a95ad14

Commits on Feb 28, 2026

[8/8] fix: CUDA IPC incompatibility from Megatron bump (#11)

guapisolo
and
claude
committed
57784b8
fix: dp_reshardable checkpoint backward compat in Megatron core

guapisolo
and
claude
committed
07220d2

Commits on Mar 4, 2026

Upgrade Megatron from Dec 17 to Feb 13

guapisolo
and
yueming-yuan
committed
038e8e5

Commits on Mar 11, 2026

Commits on Mar 13, 2026

[fix] fix mixed precision parameters load from checkpoint (#19)

yueming-yuan
and
fzyzcjy
authored
cf014f9
[feat] distributed optimizer low GPU memory resume from checkpoint (#20)

yueming-yuan
and
fzyzcjy
authored
90f69fa

Commits on Apr 14, 2026

[fix] Add detach() to fp32 param shard for leaf-tensor consistency
guapisolo
authored
32dbe9f

Commits on Apr 15, 2026

[fix] attention_output_gate TP slice when num_kv_heads < TP (#22)
Zhichenzzz
authored
923b85b

Commits on Apr 20, 2026

[fix] Enforce param dtype before wrap ddp (#24)
guapisolo
authored
d9a5080

Commits on May 18, 2026

[feat] Init true on policy with qwen_dense

authored
23924a0

Commits on May 30, 2026

Package miles_megatron_plugins with Megatron Core (#49)
maocheng23
authored
3a60706

Commits on Jun 1, 2026

Broaden megatron-core packages.find to megatron* (namespaces), keep plugins (#50)
Shi-Dong
authored
a381c7d

Commits on Jun 8, 2026

DeepSeek V4 RL support (#28)

yueming-yuan
and
fzyzcjy
authored
ae20f41
Skip missing expert bias updates (#55)
yueming-yuan
authored
36ceb4f

Commits on Jun 18, 2026

[AMD] Fix QuantizedTensor import for ROCm TransformerEngine 2.8 (#58)

XinyuJiangCMU
and
JessicaJiang-123
authored
3a8f0c7

Commits on Jun 19, 2026

fix(dist-ckpt): handle bare BytesIO _extra_state in key recovery (TE FP8) (#53)
Zhichenzzz
authored
87d1155

Commits on Jun 29, 2026

Restore enforce_marked_param_dtypes after model materialization (#60)

authored
79fc089

Commits on Jul 9, 2026

Wire witness support inside Megatron (#21)
fzyzcjy
authored
4716f75

Commits on Jul 28, 2026

feat(optimizer): NVMe streaming of DistributedOptimizer state (#63)

yueming-yuan
and
Zhichenzzz
authored
9fc14d8

Commits on Jul 30, 2026

[tml] fp32 MoE activation and combine for bf16 inference alignment (#68)
Zhichenzzz
authored
50ac48e

Commits on Aug 5, 2026

Extend FP32 MoE numerics to low precision
zianglih
committed
b70305a
Add extensible FP32 MoE activations
zianglih
committed
79f54ee
Add low-precision FP32 MoE smoke coverage
zianglih
committed
78171a9