[Dev] Hybrid Data x Context Parallelism Feature - #2054

Merged

pablo-garay merged 90 commits into

parthmannan:pmannan/hybrid_dp_cp_feature

Dec 4, 2025

Commits on Jul 15, 2025

Start heterogeneous CP idea prototyping
parthmannan
committed
d50a125

Commits on Jul 16, 2025

Add current_cp_assignment and some supporting code
parthmannan
committed
676f0bf
Add bricks for building heterogeneous CP schedules
parthmannan
committed
65a24cc

Commits on Jul 17, 2025

Return sub-sample per CP rank count to schedules
parthmannan
committed
0c5fa39

Commits on Jul 18, 2025

Move wrapper to utils and formatting
parthmannan
committed
8546b0f
Add TransformerConfig and arguments
parthmannan
committed
b17a2a2

Commits on Jul 19, 2025

Fix implementation issues [WIP]
parthmannan
committed
98f592a

Commits on Jul 21, 2025

Commits on Jul 22, 2025

Add sharding logic and TODO for sync across multiple microbatches
parthmannan
committed
056055c

Commits on Aug 13, 2025

Merge branch 'main' of https://gitlab-master.nvidia.com/ADLR/megatron-lm into pmannan/hetero_cp_support
parthmannan
committed
f069e0d

Commits on Aug 14, 2025

Updates to working milestone 1 Hybrid CP scheduling
parthmannan
committed
8d34369

Commits on Aug 21, 2025

Add SFT dataset support
parthmannan
committed
771fc21

Commits on Aug 22, 2025

New updates to use DataIterator wrapper
parthmannan
committed
3a7e50a
Test scripts
parthmannan
committed
bd2321d
Minor fixes
parthmannan
committed
6a1ab4f
SFT Dataset + GPT with Hybrid CP working example
parthmannan
committed
e01583a

Commits on Aug 23, 2025

Remove new SFT loss calculation in favor of regular loss calc
parthmannan
committed
286129c

Commits on Aug 29, 2025

Milestone 2: Add global batch scheduling, data movement and training
parthmannan
committed
2674be8

Commits on Sep 15, 2025

Fix for 0 seq len in data
parthmannan
committed
3b87385

Commits on Sep 16, 2025

Fix fill_empty_gpus workload assignment
parthmannan
committed
bd04cfe
llama3 8B benchmarking script
parthmannan
committed
160f514
Bug fix
parthmannan
committed
b0eb325
Merge branch 'main' of ssh://gitlab-master.nvidia.com:12051/ADLR/megatron-lm into pmannan/hetero_cp_test_sft
parthmannan
committed
eccf31d
Revert "ADLR/megatron-lm!3963 - SFT chat template and pad token changes for Nano v2 12B"
parthmannan
committed
7e9fe90
Turn on legacy tokenizer
parthmannan
committed
b635d15
chore: Format files
Mcore Bot
committed
dc8139f
Remove scheduled id and other finished TODO items
parthmannan
committed
69ef399
Merge branch 'pmannan/hetero_cp_test_sft' of ssh://gitlab-master.nvidia.com:12051/ADLR/megatron-lm into pmannan/hetero_cp_test_sft
parthmannan
committed
b37e3f7
Formatting and cleanup
parthmannan
committed
3bafa2c

Commits on Sep 17, 2025

Add copyright
parthmannan
committed
78dec32
Start testing
parthmannan
committed
6b1d9c4
chore: Format files
Mcore Bot
committed
71c2966

Commits on Sep 18, 2025

Fix arg name in test
parthmannan
committed
bff834c
Merge branch 'pmannan/hetero_cp_test_sft' of ssh://gitlab-master.nvidia.com:12051/ADLR/megatron-lm into pmannan/hetero_cp_test_sft
parthmannan
committed
2860324
Fix arg name in test
parthmannan
committed
ff5600b
Update RoPE logic to handle none packed_seq_params
parthmannan
committed
7ff37e5
Fix test import
parthmannan
committed
3c1f66a
chore: Format files
Mcore Bot
committed
aa6c7f7

Commits on Sep 23, 2025

fix some bugs and support TP
xiaoyao0115
committed
9d761cc
fix pad issue, should pad for sequence parallel when TP>1 and consider hdp
xiaoyao0115
committed
7d6c2b1

Commits on Sep 24, 2025

little adjustment according to comments
xiaoyao0115
committed
fbdfe18

Commits on Sep 25, 2025

little fix to loop parameter
xiaoyao0115
committed
8e6aa49

Commits on Sep 26, 2025

disable cudagraph anb fsdp when using hybrid cp
xiaoyao0115
committed
f050f54

Commits on Sep 29, 2025

Add cp comm type to dynamic CP attn
parthmannan
committed
1e76e8e
Merge branch 'main' of ssh://gitlab-master.nvidia.com:12051/ADLR/megatron-lm into pmannan/hetero_cp_test_sft
parthmannan
committed
ab8dc81
Cleanup
parthmannan
committed
3a39a3a

Commits on Oct 2, 2025

Removing benchmarking scripts for MR
parthmannan
committed
cb72e4e
Merge branch 'main' of ssh://gitlab-master.nvidia.com:12051/ADLR/megatron-lm into pmannan/hybrid_dp_cp_feature
parthmannan
committed
aeb9e36
Restore SFT dataset/tokenizer to main
parthmannan
committed
6aebf37
Update pre-train script
parthmannan
committed
f5c0e76
Lint change
parthmannan
committed
9d96178
chore: Format files
Mcore Bot
committed
b7aff32

Commits on Oct 3, 2025

Add assert
parthmannan
committed
994e65f
Merge branch 'pmannan/hybrid_dp_cp_feature' of ssh://gitlab-master.nvidia.com:12051/ADLR/megatron-lm into pmannan/hybrid_dp_cp_feature
parthmannan
committed
f60bfdd
Add support for Yarn Rope
parthmannan
committed
aa6cf10
Add support for MultimodalRope
parthmannan
committed
f3a6e17
General and RoPE support cleanup
parthmannan
committed
a0fb848
Data sampler cleanup
parthmannan
committed
b0b0fab
chore: Format files
Mcore Bot
committed
335994c
Cleanup hybrid_cp_schedule
parthmannan
committed
ba99cad
lint fixes
parthmannan
committed
d3c0c03
Fix MLA + Yarn calls
parthmannan
committed
1712980
chore: Format files
Mcore Bot
committed
62817e8

Commits on Oct 9, 2025

Commits on Oct 16, 2025

Address comments
parthmannan
committed
627753b
Lint
parthmannan
committed
a3b5ebe
chore: Format files
Mcore Bot
committed
1c22f5c

Commits on Oct 30, 2025

Merge branch 'dev' of https://github.com/NVIDIA/Megatron-LM into pmannan/hybrid_dp_cp_feature
parthmannan
committed
9c75535

Commits on Nov 7, 2025

Merge branch 'dev' of github.com:NVIDIA/Megatron-LM into pmannan/hybrid_dp_cp_feature
parthmannan
committed
97d8224

Commits on Nov 8, 2025

Fix import
parthmannan
committed
617fb17

Commits on Nov 17, 2025

Move Dataloader wrapper to core/datasets
parthmannan
committed
e95ae83
Formatting
parthmannan
committed
0a0339a
Fix conflict
parthmannan
committed
4af66dd

Commits on Nov 25, 2025

Commits on Dec 1, 2025

Commits on Dec 3, 2025

Mamba script fix
parthmannan
committed
66404ca