[feat] Generalized Tensor Parallelism (GTP) - #4967

Merged

fanshiqing merged 100 commits into

NVIDIA:mainfrom

fanshiqing:gtp_release

Jul 25, 2026

Commits on May 25, 2026

Generalized Tensor Parallelism (GTP) init commit

fanshiqing
and
jiemingz
committed
162217c
fix conflicts
fanshiqing
committed
1cab66a
code clean
fanshiqing
committed
ccfee04
fix comments; defer wgrad->dgrad support in following up MR;
fanshiqing
committed
8b4041a

Commits on May 26, 2026

fix comments
fanshiqing
committed
1903598

Commits on May 27, 2026

code clean.
fanshiqing
committed
9eb5007
Fix GTP broadcast_params + add partial DP-CP with GTP group
fanshiqing
committed
019536f
fix none-egpt sharded params's reduction in moe layer; fix comments
fanshiqing
committed
375f09c

Commits on May 28, 2026

Commits on May 29, 2026

fix comments
fanshiqing
committed
cdf5d35

Commits on Jun 1, 2026

Commits on Jun 4, 2026

gtp+gmm-fusion: support offloading(moe-act-input)
fanshiqing
committed
392816a
GTP + full-iter CG
fanshiqing
committed
fc570d0
[feat]GTP: prefetch recompute-forward weight gathers via a separate chain; update README
fanshiqing
committed
23ed3ba
GTP: allocate GRAPHED buffers into CG mempool at creation; fix comments
fanshiqing
committed
ecf2dd1
fix comments
fanshiqing
committed
1163b4a
fix onlince checks: copyright, intallation test, build.
fanshiqing
committed
5dc0423

Commits on Jun 5, 2026

fix te min version required for GTP.
fanshiqing
committed
ab869b9
fix online UTs; fix comments.
fanshiqing
committed
45a604f
fix UTs
fanshiqing
committed
87a50cd

Commits on Jun 6, 2026

Commits on Jun 8, 2026

add integration test for {mamba,attn,moe}+gtp; polish existing gtp and tp+gtp UTs.
fanshiqing
committed
4d1e2eb

Commits on Jun 10, 2026

fix comments from Jimmy and Deepak
fanshiqing
committed
4695217

Commits on Jun 11, 2026

feat: make (E)GTP a first-class orthogonal parallelism axis
fanshiqing
committed
3725f51
code clean
fanshiqing
committed
84e6a7d

Commits on Jun 12, 2026

Merge remote-tracking branch 'adlr_github/main' into gtp_release
fanshiqing
committed
42fd06f
code clean
fanshiqing
committed
b8b078a
Generate the DDP param layerout for the GTP replicate group at it's source
fanshiqing
committed
83bea9f
fold the GTP intra DP groups into intra_dp_cp_group and intra_expt_dp_group
fanshiqing
committed
22fc6e6
fix UTs
fanshiqing
committed
0a55ed5

Commits on Jun 15, 2026

[feat] GTP+DCP
fanshiqing
committed
be22dce
rename gtp-exclude process group: with_gtp -> no_gtp
fanshiqing
committed
70ef35d
fix dense GTP NCCL group using stale 'ps' key
fanshiqing
committed
e430807

Commits on Jun 17, 2026

update README with scalability
fanshiqing
committed
5a8c469

Commits on Jun 18, 2026

fix comments
fanshiqing
committed
aa40d0d

Commits on Jun 19, 2026

fix comments
fanshiqing
committed
601a658
Rename GTP remat knobs and add num-weight-shards user API
fanshiqing
committed
eddb7ba

Commits on Jun 20, 2026

Support GTP/EGTP in LayerWiseDistributedOptimizer and Muon (#3)

authored and
fanshiqing
committed
6806f43
GTP+DCP: simplify gtp replica_ids in MambaMixer.sharded_state_dict; add corresponding UTs
fanshiqing
committed
7d71c08
GTP+Muon: fix DCP save/load; add corresponding UTs
fanshiqing
committed
69dae5a
code clean and fix comments
fanshiqing
committed
f13a042

Commits on Jun 23, 2026

Fix GTP DDP grad-ready firing before deferred wgrad accumulation
fanshiqing
committed
0e3c3d2

Commits on Jun 24, 2026

fix format and comments
fanshiqing
committed
f6dca05

Commits on Jun 25, 2026

Merge remote-tracking branch 'adlr_github/main' into gtp_release
fanshiqing
committed
c33667a
fix comments
fanshiqing
committed
ce02728
fix linting
fanshiqing
committed
ae8a571
Merge remote-tracking branch 'adlr_github/main' into gtp_release
fanshiqing
committed
71a53c6
Fix optional process-group fallbacks defeated by __getattr__; Log human-readable dtype in GTP weight cache
fanshiqing
committed
6aeecc1
Fix GTP grad norm inflated on CUDA-graph capture step; fix linting
fanshiqing
committed
114b6fc
fix online UTs
fanshiqing
committed
3c7aa6c

Commits on Jun 26, 2026

Commits on Jun 27, 2026

Commits on Jun 28, 2026

Commits on Jun 29, 2026

Refactor: make GTP_remat truly independent from DP
fanshiqing
committed
5b314c2

Commits on Jun 30, 2026

Make get_data_parallel_group default to the gtp_remat-inclusive axis
fanshiqing
committed
e2f9374
Fix: skip GTP all-gather for non-GTP params in Muon orthogonalization (#6)

deepakn94
authored and
fanshiqing
committed
ee49c53
fix formating
fanshiqing
committed
0a2581a

Commits on Jul 2, 2026

Commits on Jul 6, 2026

Commits on Jul 7, 2026

Merge remote-tracking branch 'adlr_github/main' into gtp_release
fanshiqing
committed
cbde2b0
code clean and fix after merge
fanshiqing
committed
934e7d9

Commits on Jul 9, 2026

Commits on Jul 10, 2026

fix: skip GTP_remat grad reduction when the axis is inactive
fanshiqing
committed
f359ef9

Commits on Jul 13, 2026

GTP+nvfp4: enforce --fp4-param-gather for GTP under --fp4-format
fanshiqing
committed
448915e
Implement TE DistributedWeight protocol on GTPShardedParam
fanshiqing
committed
29d49d9
Fix GTP grad state after partial CUDA graph replay (#10)

JF-D
authored and
fanshiqing
committed
ccfed80

Commits on Jul 14, 2026

Update GTP document
fanshiqing
committed
bff85a6

Commits on Jul 15, 2026

Track TE NVFP4 gather API changes in the GTP call site
fanshiqing
committed
255e6e4

Commits on Jul 16, 2026

Support Muon (LayerWiseDistributedOptimizer) with mxfp8 fp8-param-gather
fanshiqing
committed
1b0121d

Commits on Jul 17, 2026

Make GTP self-sufficient for the DistributedWeight contract
fanshiqing
committed
b5359b3

Commits on Jul 20, 2026

Commits on Jul 24, 2026

Commits on Jul 25, 2026

Merge branch 'main' into gtp_release
fanshiqing
committed
d6af075
Fix distributed-optimizer shard count with multiple optimizer instances
fanshiqing
committed
19f8482