Code

enable LoRA + FSDP2#855

Code

Merged

weifengpy merged 58 commits intopytorch:mainpytorch/torchtune:mainfrom weifengpy:fsdp2weifengpy/torchtune:fsdp2Copy head branch name to clipboardJun 3, 2024

+1,493-4

Commits

Commits on Apr 24, 2024

• enable LoRA + FSDP2

  Show description for e5826a1
  weifengpy

  committed
  e5826a1

  Copy full SHA for e5826a1

• reset params for lora weights and rope

  Show description for 64fc870
  weifengpy

  committed
  64fc870

  Copy full SHA for 64fc870

• support lora weights checkpoint and checkpoint utils

  Show description for 0cd21c6
  weifengpy

  committed
  0cd21c6

  Copy full SHA for 0cd21c6

• fix lora meta device bug

  Show description for 589191e
  weifengpy

  committed
  589191e

  Copy full SHA for 589191e

Commits on Apr 25, 2024

• save optim state dict

  Show description for c801f26
  weifengpy

  committed
  c801f26

  Copy full SHA for c801f26

• mark TODO

  Show description for 19a2d70
  weifengpy

  committed
  19a2d70

  Copy full SHA for 19a2d70

• optimizer foreach=True for DTensor

  Show description for 441da10
  weifengpy

  committed
  441da10

  Copy full SHA for 441da10

• clip grad norm

  Show description for 750b9e5
  weifengpy

  committed
  750b9e5

  Copy full SHA for 750b9e5

Commits on Apr 26, 2024

• switch to ptd state dict api

  Show description for 3d632d5
  weifengpy

  committed
  3d632d5

  Copy full SHA for 3d632d5

Commits on May 1, 2024

• add profiler

  Show description for cb3abb3
  weifengpy

  committed
  cb3abb3

  Copy full SHA for cb3abb3

• use torchao copy_

  Show description for e68804a
  weifengpy

  committed
  e68804a

  Copy full SHA for e68804a

• enable saving checkpoint

  Show description for d6af9a2
  weifengpy

  committed
  d6af9a2

  Copy full SHA for d6af9a2

• optimizer state dict: load on rank0 and broadcast

  Show description for b616394
  weifengpy

  committed
  b616394

  Copy full SHA for b616394

• import Optimizer

  Show description for a400497
  weifengpy

  committed
  a400497

  Copy full SHA for a400497

Commits on May 3, 2024

• resume training

  Show description for e9de63c
  weifengpy

  committed
  e9de63c

  Copy full SHA for e9de63c

• prepare for full test

  Show description for 05d3895
  weifengpy

  committed
  05d3895

  Copy full SHA for 05d3895

• prepare for full test

  Show description for 7a5bb80
  weifengpy

  committed
  7a5bb80

  Copy full SHA for 7a5bb80

• remove profiler

  Show description for 64bf49c
  weifengpy

  committed
  64bf49c

  Copy full SHA for 64bf49c

Commits on May 4, 2024

• passed integration test

  Show description for cb1bba4
  weifengpy

  committed
  cb1bba4

  Copy full SHA for cb1bba4

• remove uncesssary change

  Show description for ac516e9
  weifengpy

  committed
  ac516e9

  Copy full SHA for ac516e9

• Merge branch 'main' into fsdp2

  weifengpy

  authored
  bfde704

  Copy full SHA for bfde704

• bring back state dict validation

  Show description for 102db31
  weifengpy

  committed
  102db31

  Copy full SHA for 102db31

• align indent on comment

  Show description for 0b66651
  weifengpy

  committed
  0b66651

  Copy full SHA for 0b66651

• remove unused import

  Show description for 672aabb
  weifengpy

  committed
  672aabb

  Copy full SHA for 672aabb

Commits on May 8, 2024

• switch to ptd state dict and keep self implemented in record

  Show description for 6af2723
  weifengpy

  committed
  6af2723

  Copy full SHA for 6af2723

• clean unused code

  Show description for 42ad99c
  weifengpy

  committed
  42ad99c

  Copy full SHA for 42ad99c

• remove cuda value error

  Show description for 74f6175
  weifengpy

  committed
  74f6175

  Copy full SHA for 74f6175

• comment on to_empty

  Show description for f1b8a5e
  weifengpy

  committed
  f1b8a5e

  Copy full SHA for f1b8a5e

• fix memory issues by switching model state dict api

  Show description for 36e6829
  weifengpy

  committed
  36e6829

  Copy full SHA for 36e6829

• clean for review

  Show description for 08cd1fd
  weifengpy

  committed
  08cd1fd

  Copy full SHA for 08cd1fd

• Merge branch 'main' into fsdp2

  weifengpy

  authored
  559bc4d

  Copy full SHA for 559bc4d

Commits on May 9, 2024

• fix linter

  Show description for 2333134
  weifengpy

  committed
  2333134

  Copy full SHA for 2333134

• fix checkpoint loading

  Show description for 49a0364
  weifengpy

  committed
  49a0364

  Copy full SHA for 49a0364

• expecttest CI depedency

  Show description for dc2ce02
  weifengpy

  committed
  dc2ce02

  Copy full SHA for dc2ce02

• ci depdencecy

  Show description for 0a604aa
  weifengpy

  committed
  0a604aa

  Copy full SHA for 0a604aa

Commits on May 10, 2024

• fix CI issue

  Show description for fa83140
  weifengpy

  committed
  fa83140

  Copy full SHA for fa83140

• Merge branch 'pytorch:main' into fsdp2

  weifengpy

  authored
  4b5a895

  Copy full SHA for 4b5a895

Commits on May 14, 2024

• support resuming training

  Show description for a2e34ec
  weifengpy

  committed
  a2e34ec

  Copy full SHA for a2e34ec

• update docstring

  Show description for 6142031
  weifengpy

  committed
  6142031

  Copy full SHA for 6142031

• remove depdency on broadcast_from_rank0

  Show description for 7607e14
  weifengpy

  committed
  7607e14

  Copy full SHA for 7607e14

Commits on May 15, 2024

• remove the need for model.to(device)

  Show description for 1899beb
  weifengpy

  committed
  1899beb

  Copy full SHA for 1899beb

Commits on May 17, 2024

• wrap lora and TransformerBlock

  Show description for c1cfabb
  weifengpy

  committed
  c1cfabb

  Copy full SHA for c1cfabb

• require torch version 2.4.0

  Show description for d7382ae
  weifengpy

  committed
  d7382ae

  Copy full SHA for d7382ae

Commits on May 22, 2024

• FSDP(CheckpointWrapper(model))

  Show description for d1ff53b
  weifengpy

  committed
  d1ff53b

  Copy full SHA for d1ff53b

Commits on May 29, 2024

• remove model.to()

  Show description for 1eb9e87
  weifengpy

  committed
  1eb9e87

  Copy full SHA for 1eb9e87

Commits on May 31, 2024

• add docstrings and remove depdency on dcp

  Show description for 695e959
  weifengpy

  committed
  695e959

  Copy full SHA for 695e959

Commits on Jun 1, 2024

• remove try...catch FSDPModule

  Show description for e10f638
  weifengpy

  committed
  e10f638

  Copy full SHA for e10f638

• Merge branch 'main' into fsdp2

  weifengpy

  authored
  b1e3d30

  Copy full SHA for b1e3d30

• fsdp2 as dev recipe

  Show description for 944a723
  weifengpy

  committed
  944a723

  Copy full SHA for 944a723

• restore lora_finetune_distributed

  Show description for ac5f7aa
  weifengpy

  committed
  ac5f7aa

  Copy full SHA for ac5f7aa

Commits on Jun 2, 2024

• test cudnn ci error

  Show description for d769626
  weifengpy

  committed
  d769626

  Copy full SHA for d769626

Commits on Jun 3, 2024

• test CI error

  Show description for f90c3cc
  weifengpy

  committed
  f90c3cc

  Copy full SHA for f90c3cc

• address CI error for setting seed

  Show description for 42ef49a
  weifengpy

  committed
  42ef49a

  Copy full SHA for 42ef49a

• add back pytest

  Show description for 170de94
  weifengpy

  committed
  170de94

  Copy full SHA for 170de94

• add expecttest

  Show description for f8a7018
  weifengpy

  committed
  f8a7018

  Copy full SHA for f8a7018

• pytest 7.4.0

  Show description for a3b2f3e
  weifengpy

  committed
  a3b2f3e

  Copy full SHA for a3b2f3e

• add dev/recipe

  Show description for 1a692b3
  weifengpy

  committed
  1a692b3

  Copy full SHA for 1a692b3

• update yaml with lora_finetune_fsdp2

  Show description for 8fbbc4b
  weifengpy

  committed
  8fbbc4b

  Copy full SHA for 8fbbc4b