Repository navigation

🚨 TP dtensor API inference + training - #47579

Merged

3outeille merged 244 commits into

mainfrom

split/b-pr-0-dual-path-tp

Aug 21, 2026

Merged

🚨 TP dtensor API inference + training#47579
3outeille merged 244 commits into
mainfrom
split/b-pr-0-dual-path-tp

Commits on Jun 17, 2026
- add distributed config
  3outeille
  committed
  799ac94
- Add native FSDP2 module and migrate FSDP imports (Phase A PR-2).
  3outeille
  committed
  22d4b52
- linting
  3outeille
  committed
  4bfd1a6
- unecessary
  3outeille
  committed
  9487bdd
- copyright edit
  3outeille
  committed
  588884e
- revert
  3outeille
  committed
  8cc48a0
- add shard on read
  3outeille
  committed
  672bbbe
Commits on Jun 23, 2026
- Merge branch 'main' into split/a-pr-1-distributed-config
  3outeille
  authored
  5bbc796
- jsut shard on read
  3outeille
  committed
  d0cc6a7
- Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp-module
  3outeille
  authored
  6fd7813
- Merge branch 'main' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  47519d7
- cleaning
  3outeille
  committed
  063e57b
- Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/huggingface/transformers into split/a-pr-3-dual-path-loading
  3outeille
  committed
  c06dc35
- linting
  3outeille
  committed
  9d2e86d
- fix
  3outeille
  committed
  79457b3
- fix
  3outeille
  committed
  f3e8021
- Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp-module
  3outeille
  authored
  acacae8
- Merge branch 'main' into split/a-pr-1-distributed-config
  3outeille
  authored
  54c1f4e
- Merge branch 'main' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  d23986a
- Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp-module
  3outeille
  authored
  f219c74
Commits on Jun 24, 2026
- remove redundant test file
  3outeille
  committed
  ea8243f
- Merge branch 'main' into split/a-pr-1-distributed-config
  3outeille
  authored
  db31b04
- Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp-module
  3outeille
  authored
  4d840dc
- Update src/transformers/distributed/fsdp.py
  
  3outeille
  and
  ArthurZucker
  authored
  c384fcd
- avoid looping, just look at dict
  3outeille
  committed
  9625816
- expand_fsdp returns reshard_targets, no_reshard_targets right away
  3outeille
  committed
  59bcec5
- better _resolve_tied_embed_lm_head_plan
  3outeille
  committed
  ebf3585
- cleaning
  3outeille
  committed
  e969325
- ruff
  3outeille
  committed
  2376965
- Merge branch 'main' into split/a-pr-1-distributed-config
  3outeille
  authored
  d830114
- Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp-module
  3outeille
  authored
  a44f81f
- more robust detection of embed and lm_head
  3outeille
  committed
  0f62c45
- Merge branch 'split/a-pr-2-fsdp-module' of https://github.com/huggingface/transformers into split/a-pr-2-fsdp-module
  3outeille
  committed
  020f7d3
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  05d766f
- cleaning
  3outeille
  committed
  da302ad
- ruff
  3outeille
  committed
  dfc665c
- typo
  3outeille
  committed
  446fd6e
- cleaner
  3outeille
  committed
  5aeaff7
- cleaner
  3outeille
  committed
  819ff14
- Merge branch 'main' into split/a-pr-1-distributed-config
  3outeille
  authored
  413d775
- Merge branch 'split/a-pr-1-distributed-config' into split/a-pr-2-fsdp-module
  3outeille
  authored
  7bc3722
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  d6d8b51
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  f8f27ff
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  ce2f001
Commits on Jun 25, 2026
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  ec87fff
Commits on Jun 26, 2026
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  6e156fd
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  606df0a
Commits on Jun 29, 2026
- typo
  3outeille
  committed
  3ee8a92
- Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/huggingface/transformers into split/a-pr-3-dual-path-loading
  3outeille
  committed
  6bc43ae
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  c4aa4b7
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  c5ad67b
- refactor dense path + apply_contiguous_shard
  3outeille
  committed
  3c6c5f2
- linting
  3outeille
  committed
  74490d6
- cleaning
  3outeille
  committed
  db6c99e
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  04c124e
- refactor _apply_strided_shard
  3outeille
  committed
  8f214ef
- better
  3outeille
  committed
  ab34afb
- refactor _slice_and_cat
  3outeille
  committed
  c4bf166
- better comment
  3outeille
  committed
  be61c49
Commits on Jul 1, 2026
- refactor moe dtensor shard ops
  3outeille
  committed
  75ad5b9
- better comment
  3outeille
  committed
  91cd797
- comment
  3outeille
  committed
  68069de
- cleaning
  3outeille
  committed
  83029ed
- linting
  3outeille
  committed
  1f857df
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  aa140ed
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  8b57aa4
- Add FSDP orchestration: mesh init, distribute-before-load, and DCP save.
  3outeille
  committed
  7c11394
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  70544cd
- Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-orchestration
  3outeille
  authored
  17c6d40
- add fsdp plan to 2 models for now
  3outeille
  committed
  00eb116
- add tests fsdp mixin
  3outeille
  committed
  be296dd
- linting
  3outeille
  committed
  05900d6
- refactor test fsdp mixin
  3outeille
  committed
  fc2423b
- test fsdp mixin cleaning
  3outeille
  committed
  5bbd820
- remove fsdp policy in tests + trim down further
  3outeille
  committed
  b6d0b67
- test fsdp clean
  3outeille
  committed
  ea36123
- restore test_modeling_utils
  3outeille
  committed
  bec4d23
- linting
  3outeille
  committed
  8d3d329
- start trim down stuff
  3outeille
  committed
  6316ee1
- fix
  3outeille
  committed
  6e9004e
Commits on Jul 2, 2026
- breaking: cleaning modeling_utils.py
  3outeille
  committed
  68df491
- load path with fsdp (dtensor) and tp (old tp) is linked
  3outeille
  committed
  16b0b29
- linting
  3outeille
  committed
  e976a44
- add saving
  3outeille
  committed
  a2fb155
- styling
  3outeille
  committed
  5f52f19
Commits on Jul 3, 2026
- fix tp ci
  3outeille
  committed
  7f54301
- add fsdp to ci
  3outeille
  committed
  99f79ac
- linting
  3outeille
  committed
  b451490
- pick one model only for this PR
  3outeille
  committed
  54ff4d1
- restore
  3outeille
  committed
  3399539
- trigger fsdp ci
  3outeille
  committed
  11cf79c
- doc cleaning + tp_size remove
  3outeille
  committed
  5b7ac3e
- fix tp ci for ep
  3outeille
  committed
  06b0c39
- edit doc
  3outeille
  committed
  7a94d77
- move distributed function to utils + guarding
  3outeille
  committed
  f3c742b
- linting
  3outeille
  committed
  37df13e
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  dd1000b
- expand_fsdp_plan iterate over modules
  3outeille
  committed
  5c95559
- Merge branch 'split/a-pr-2-fsdp-module' of https://github.com/huggingface/transformers into split/a-pr-2-fsdp-module
  3outeille
  committed
  00a11b6
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  fd3d4b0
- Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-orchestration
  3outeille
  authored
  86875d2
- comment about tie embedding
  3outeille
  committed
  e4613e6
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  c2b8451
- add comment tied embedding
  3outeille
  committed
  d558f99
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  aed1818
- Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-orchestration
  3outeille
  authored
  450579b
- add DistributedMixin
  3outeille
  committed
  91260ba
Commits on Jul 6, 2026
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  1541674
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  9fd759d
- solve merge conflict
  3outeille
  committed
  84dd107
Commits on Jul 7, 2026
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  9fae1b4
- some cleaning
  3outeille
  committed
  eb653bd
- Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/huggingface/transformers into split/a-pr-3-dual-path-loading
  3outeille
  committed
  dcaa20a
- cleaning + comment
  3outeille
  committed
  d03499f
- rename function for clarity
  3outeille
  committed
  fb5533e
- Apply suggestion from @ArthurZucker
  
  3outeille
  and
  ArthurZucker
  authored
  7838dc6
- doc
  3outeille
  committed
  877052b
- Merge branch 'split/a-pr-3-dual-path-loading' of https://github.com/huggingface/transformers into split/a-pr-3-dual-path-loading
  3outeille
  committed
  59f666f
- comment
  3outeille
  committed
  17643a7
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  7bb152c
Commits on Jul 8, 2026
- Merge branch 'main' into split/a-pr-2-fsdp-module
  3outeille
  authored
  0b4d1e0
- Merge branch 'split/a-pr-2-fsdp-module' into split/a-pr-3-dual-path-loading
  3outeille
  authored
  101562d
- Merge branch 'split/a-pr-3-dual-path-loading' into split/a-pr-4-fsdp-orchestration
  3outeille
  authored
  99576ee
- linting
  3outeille
  committed
  22215b7
- Merge branch 'split/a-pr-4-fsdp-orchestration' of https://github.com/huggingface/transformers into split/a-pr-4-fsdp-orchestration
  3outeille
  committed
  f6ed465
- refactor
  3outeille
  committed
  18d709c
- abstract to mixin
  3outeille
  committed
  566ef6e
- typo
  3outeille
  committed
  b632db3
- Add FSDP plans to all models from distributed branch.
  3outeille
  committed
  0eec8c7
- fsdp plans
  3outeille
  committed
  a1f4b71
- add fsdp plans
  3outeille
  committed
  a07e5bf
- linting
  3outeille
  committed
  0dfe224
Commits on Jul 9, 2026
- linting
  3outeille
  committed
  e5a6535
Commits on Jul 16, 2026
- Add distributed runtime utils and DistributedMixin (FSDP orchestration 1/3).
  3outeille
  committed
  e764ceb
- Wire DistributedConfig through from_pretrained and save_pretrained (FSDP orchestration 2/3).
  3outeille
  committed
  61362b7
- Add FSDP CI and end-to-end FSDP tests (FSDP orchestration 3/3).
  3outeille
  committed
  eb34d37
- addd ep_plan
  3outeille
  committed
  b866f00
- Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2-orchestration
  3outeille
  authored
  3da6fb9
- Merge branch 'split/a-pr-4-2-orchestration' into split/a-pr-4-3-fsdp-tests-ci
  3outeille
  authored
  ca8d037
- restore validate module
  3outeille
  committed
  b3f4538
- Wire DistributedConfig through from_pretrained and save_pretrained.
  3outeille
  committed
  b1fb1b9
- revert
  3outeille
  committed
  1bf32cc
- Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2-orchestration
  3outeille
  authored
  1620c0f
- Merge branch 'split/a-pr-4-2-orchestration' into split/a-pr-4-3-fsdp-tests-ci
  3outeille
  authored
  fe0dcf7
- inline distribute_model
  3outeille
  committed
  e52fda3
- revert
  3outeille
  committed
  71dde8a
- remove saving/loading
  3outeille
  committed
  4e23dbf
- leaner mixin
  3outeille
  committed
  5742a2e
- downgrade torch version guarding
  3outeille
  committed
  5c81035
- remove
  3outeille
  committed
  004354f
- linting
  3outeille
  committed
  6d03414
- Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-3-fsdp-tests-ci
  3outeille
  authored
  450987f
- revert
  3outeille
  committed
  47d85ad
- Merge branch 'split/a-pr-4-2-fsdp-tests-ci' of https://github.com/huggingface/transformers into split/a-pr-4-2-fsdp-tests-ci
  3outeille
  committed
  c577f33
- revert
  3outeille
  committed
  b319d6a
- post_init() parallel plan move to mixin
  3outeille
  committed
  eb9a081
- Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2-fsdp-tests-ci
  3outeille
  authored
  d5358f3
- revert tp mixin
  3outeille
  committed
  9a783c8
- Merge branch 'split/a-pr-4-1-distributed-runtime' into split/a-pr-4-2-fsdp-tests-ci
  3outeille
  authored
  0448e72
- add save/load
  3outeille
  committed
  259bff4
- Merge branch 'main' into split/a-pr-4-1-distributed-runtime
  3outeille
  authored
  b15b63e
- Merge branch 'main' into split/a-pr-4-1-distributed-runtime
  3outeille
  authored
  17d611b
Commits on Jul 21, 2026
Commits on Jul 23, 2026
Commits on Jul 24, 2026
- Merge branch 'main' into split/a-pr-5-fsdp-plans
  3outeille
  authored
  c364a54
- ea
  3outeille
  committed
  36ca137
- Merge branch 'main' into split/a-pr-5-fsdp-plans
  3outeille
  authored
  2f87b7d
Commits on Jul 27, 2026
- begin migration TP
  3outeille
  committed
  a171a19
- clean up
  3outeille
  committed
  71255bf
- Merge branch 'main' into split/a-pr-5-fsdp-plans
  3outeille
  authored
  d44d5db
- add it to pretrained model
  3outeille
  committed
  507a3c7
- linting
  3outeille
  committed
  258ab19
- fix the test by moving in init class the fsdp plan instead of post init
  3outeille
  committed
  70de295
- edit
  3outeille
  committed
  830f0e4
- replace everything
  3outeille
  committed
  0ddc172
- Merge branch 'split/a-pr-5-fsdp-plans' into distributed_guarding
  3outeille
  authored
  7193907
- Merge branch 'distributed_guarding' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  4cb403f
- begin migration TP
  3outeille
  committed
  ad7248f
- clean up
  3outeille
  committed
  1dcd5d2
- revert merge conflict
  3outeille
  committed
  2cccb89
- local params for forward
  3outeille
  committed
  4098e00
- merged
  3outeille
  committed
  ed5c136
- cleaning
  3outeille
  committed
  b972782
Commits on Jul 28, 2026
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  2469b2b
Commits on Jul 29, 2026
- migration from integration.tensor_parallel to distributed.tensor_parallel
  3outeille
  committed
  c416695
- xMerge branch 'split/b-pr-0-dual-path-tp' of github.com:huggingface/transformers into split/b-pr-0-dual-path-tp
  3outeille
  committed
  68e932d
- revert models
  3outeille
  committed
  673188f
- revert
  3outeille
  committed
  d2ecb9b
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  7aacfdf
- fix
  3outeille
  committed
  5f392ff
- breaking: cleaner way to_local for quantize weight(almost done need to fix backward)
  3outeille
  committed
  a2aaec7
Commits on Jul 30, 2026
- fix all tests
  3outeille
  committed
  2119865
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  48bf5b5
Commits on Aug 3, 2026
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  911ac80
- TP dtensor handle natively the to_local() for deepgem + fp8 (#47634)
  3outeille
  authored
  42edef6
- bench: dtensor vs legacy TP
  3outeille
  committed
  dec0187
Commits on Aug 5, 2026
Commits on Aug 6, 2026
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  c10dd03
- remove dead code in mxfp4
  3outeille
  committed
  d0cdd9c
- Merge branch 'split/b-pr-0-dual-path-tp' of github.com:huggingface/transformers into split/b-pr-0-dual-path-tp
  3outeille
  committed
  02077ae
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  57cf762
- remove old TP
  3outeille
  committed
  6c4063d
- Merge branch 'split/b-pr-0-dual-path-tp' of github.com:huggingface/transformers into split/b-pr-0-dual-path-tp
  3outeille
  committed
  078f128
- cleaning
  3outeille
  committed
  8c66c74
- cleaning
  3outeille
  committed
  8b5f362
- claening
  3outeille
  committed
  8e26b06
- add test_shard_tensor_shape_consistency
  3outeille
  committed
  7febb1b
- cleaning
  3outeille
  committed
  1333432
- small fix
  3outeille
  committed
  7eaf787
- typo
  3outeille
  committed
  e62fff9
- comment
  3outeille
  committed
  2af1175
Commits on Aug 17, 2026
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  5e450f3
- rename colwise_gather_output to colwise_rep
  3outeille
  committed
  a29469e
- Merge branch 'split/b-pr-0-dual-path-tp' of github.com:huggingface/transformers into split/b-pr-0-dual-path-tp
  3outeille
  committed
  637e37a
- keep tp import to avoid breaking chnges
  3outeille
  committed
  b41ddc9
- remove async_op=True for redistribute
  3outeille
  committed
  488b801
- linting
  3outeille
  committed
  1230989
- renaming MoeExpertsParallel class
  3outeille
  committed
  1fc95b3
- ep router doc
  3outeille
  committed
  351a4cd
Commits on Aug 20, 2026
- mlinter
  3outeille
  committed
  2fd62de
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  ddced0c
- add todo
  3outeille
  committed
  2a37985
- Merge branch 'split/b-pr-0-dual-path-tp' of github.com:huggingface/transformers into split/b-pr-0-dual-path-tp
  3outeille
  committed
  4eb3126
- fix
  3outeille
  committed
  78f88a2
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  d88b9b5
- Merge branch 'main' into split/b-pr-0-dual-path-tp
  3outeille
  authored
  e2644a6
Commits on Aug 21, 2026

Commits

Commits on Jun 17, 2026

Commits on Jun 23, 2026

Commits on Jun 24, 2026

Commits on Jun 25, 2026

Commits on Jun 26, 2026

Commits on Jun 29, 2026

Commits on Jul 1, 2026

Commits on Jul 2, 2026

Commits on Jul 3, 2026

Commits on Jul 6, 2026

Commits on Jul 7, 2026

Commits on Jul 8, 2026

Commits on Jul 9, 2026