Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FSDP] Consolidate cpu_adam optimizer state dict #607

Merged
merged 2 commits into from
Apr 20, 2021
Merged

[FSDP] Consolidate cpu_adam optimizer state dict #607

merged 2 commits into from
Apr 20, 2021

Conversation

sshleifer
Copy link
Contributor

Problem

  • If an optimizer, like CPUAdam, keeps state on CPU, all_gather will break.

Approach

  • move tensor entries to cuda before all_gather

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 14, 2021
@sshleifer sshleifer added FSDP FullyShardedDataParallel (zero-3) and removed CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. labels Apr 14, 2021
@sshleifer sshleifer linked an issue Apr 14, 2021 that may be closed by this pull request
FSDP: Consolidate cpu_adam optimizer state dict #602
Closed
@sshleifer sshleifer changed the title [FSDP][Fix] [FSDP] Consolidate cpu_adam optimizer state dict Apr 14, 2021
@sshleifer sshleifer merged commit d9f3613 into master Apr 20, 2021
@sshleifer sshleifer deleted the cpu-adam-support branch April 20, 2021 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@myleott myleott myleott approved these changes

Assignees
No one assigned
Labels
FSDP FullyShardedDataParallel (zero-3)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

FSDP: Consolidate cpu_adam optimizer state dict
3 participants
@sshleifer @myleott @facebook-github-bot