[DEV] torch_dist fixes, speed improvements and memory reduction for layerwise distopt - #1965

Merged

FDecaYed merged 11 commits into

FDecaYed:deyuf/layerwise_improvements

Nov 6, 2025

Commits on Nov 5, 2025

speed and memory improvements for layerwise distopt:
FDecaYed
committed
e0fdf45
add back broadcast as reference
FDecaYed
committed
9b7f095
fix unnecessary FP32 megatron optimizer wrap and undetected ShardedTensorFactory replica_id change
FDecaYed
committed
70333ff
lint
FDecaYed
committed
8cb4004
lint
FDecaYed
committed
9fbb48e
fix forgot to remove check
FDecaYed
committed
15fc11e
fix empty group breaks common_dict after sharding
FDecaYed
committed
9f43c2f
address comments
FDecaYed
committed
325366c
fix
FDecaYed
committed
2c01a2d
fix when there is only 1 optimizer
FDecaYed
committed
ac64381

Commits on Nov 6, 2025

Merge branch 'dev' into deyuf/layerwise_improvements
FDecaYed
authored
75a42dc