[Collective][PR 4/6] NCCL Communicator caching and preliminary stream management #13030
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhisbug
changed the title
6 tasks
zhisbug
force-pushed
the
ray-collective-pr4
branch
2 times, most recently
from
33f529a to
633f188
Compare
zhisbug
changed the title
Contributor
Author
|
@richardliaw This is a big one but good to go! Now it supports any multi-GPU collective calls on actors with >1 gpus! |
richardliaw
requested changes
Jan 24, 2021
Contributor
richardliaw
left a comment
richardliaw
left a comment
There was a problem hiding this comment.
Still need to review nccl_collective_group. Will review tomorrow!
Contributor
Author
Addressed! |
richardliaw
approved these changes
Jan 26, 2021
fishbone
pushed a commit
to fishbone/ray
that referenced
this pull request
Feb 16, 2021
… management (ray-project#13030) Co-authored-by: Dacheng Li <[email protected]>
fishbone
added a commit
to fishbone/ray
that referenced
this pull request
Feb 16, 2021
…y stream management (ray-project#13030)" This reverts commit ed6b212.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the 4th PR for the project Collective-in-Ray.
See a list below:
Basic infrastructure; an in-actor collective interfaceray.util.collective.init_collective_group(*args, **kwargs); support for two collectivesallreduceandbarrier; some testing infrastructure, etc.Driver-program interface, which includes: (1) the second interface:actor.options(collective_options, ...).remote()and the third interfacedeclare_collective_group(actors, collective_options, ...).Support for other collectives:.allgather,broadcast,reduce,reducescatter; refactor the tests into distributed tests and single-node cluster tests3.5 (PR [Collective][PR 3.5/6] Send/Recv calls and some initial code for communicator caching #12935 merged)
send/recv P2P communication apis.Related issue number
See this rfc-collective-in-ray
Checks
scripts/format.shto lint the changes in this PR.