RFC: reduce allgather costs #217
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maximsch2
requested review from
ananyahjha93,
Borda,
justusschock,
SkafteNicki and
tchaton
as code owners
for more information, see https://pre-commit.ci
Codecov Report
@@ Coverage Diff @@
## master #217 +/- ##
==========================================
+ Coverage 96.72% 96.73% +0.01%
==========================================
Files 92 184 +92
Lines 2935 5888 +2953
==========================================
+ Hits 2839 5696 +2857
- Misses 96 192 +96
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
|
Actually implemented a more general solution for dist_reduce_fx="cat" and switched metrics to using it. Interestingly, we didn't have any built-in metrics using "cat" reduction - all of them were doing it manually. |
SkafteNicki
reviewed
Apr 30, 2021
Co-authored-by: Nicki Skafte <[email protected]>
|
Oh, nice, I missed those! |
SkafteNicki
approved these changes
May 3, 2021
Borda
approved these changes
May 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting
What does this PR do?
We have a number of metrics that are following a pattern of accumulating (preds, targets) in the state as a list. This leads to a lot of individual AllGather communications at the very end as we end up running them one validation batch at a time. In this diff, I'm suggesting we do a
caton those states before we do synchronization.I'm showing it only for AUROC (sending PR to get CI to run tests, but they do pass locally). If people agree that this is a good idea, we can discuss how to do this - might be we can have a flag on the base
Metricthat will enable this in vanilla_sync_distfunction (to avoid having to copy/paste this snippet around to PR, AveragePrecision and possible other metrics as well)PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃