New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use allreduce_coalesced for factor allreduce #35

Open

gpauloski opened this issue Mar 10, 2022 · 1 comment

Labels

enhancement help wanted pytorch-1.11

Owner

gpauloski commented Mar 10, 2022

pytorch/pytorch#62140

"grouped comm on a set of unflattened tensors can be more performant than flattening+a single flat nccl call."

gpauloski added the pytorch-1.11 label

Owner Author

gpauloski commented Mar 25, 2022

Can also use allgather_coalesced instead of gradient/inverse broadcast.

gpauloski added the enhancement label

gpauloski self-assigned this

gpauloski added the help wanted label

gpauloski removed their assignment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment