Skip to content

[Collective][PR 4/6] NCCL Communicator caching and preliminary stream management #13030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
richardliaw merged 35 commits into from
Jan 26, 2021
Merged

[Collective][PR 4/6] NCCL Communicator caching and preliminary stream management #13030

Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
02c6298
added cache and multi-gpu support, adding example
DachengLi1 Dec 13, 2020
2200a83
example
DachengLi1 Dec 13, 2020
fd35f72
no message
DachengLi1 Dec 14, 2020
e569bda
fix some errors
DachengLi1 Dec 14, 2020
a5003bb
multi-gpu, cache implemented
DachengLi1 Dec 15, 2020
040fc56
streams support, adding test
DachengLi1 Dec 17, 2020
e713984
fixed errors
DachengLi1 Dec 17, 2020
abfdc4d
fixed in 2x2 cases
DachengLi1 Dec 17, 2020
106330e
merge master
Dec 28, 2020
08633bf
rebase and mark a few todos
Dec 28, 2020
cc24e14
testing
DachengLi1 Dec 29, 2020
5513a08
coded 1,2, testing
DachengLi1 Dec 30, 2020
906c653
allreduce, reduce, broadcast
DachengLi1 Dec 30, 2020
06e5ea1
P2P
DachengLi1 Dec 31, 2020
2417201
allgather
DachengLi1 Dec 31, 2020
52c8a35
allgather
DachengLi1 Dec 31, 2020
308b8c1
reduce scatter
DachengLi1 Dec 31, 2020
633f188
unit testing done
DachengLi1 Dec 31, 2020
f68fd72
merge master and do a big chunk of improvement
Jan 13, 2021
77db2aa
some obvious fix
Jan 14, 2021
05a71f1
Merge branch 'master' into ray-collective-pr4
Jan 14, 2021
4ae9910
all tests run through, with some fix
zhisbug Jan 15, 2021
9364061
remove testing examples
Jan 15, 2021
33a6787
add multigpu test cases
Jan 15, 2021
1daf9c3
add allreduce_multigpu tests
zhisbug Jan 18, 2021
0e59f48
all multigpu calls works; refactor the tests so it works better now
zhisbug Jan 23, 2021
0e12bc7
auto linting
zhisbug Jan 23, 2021
6474d2a
mannual linting
zhisbug Jan 23, 2021
c661fd5
auto linting round 2
zhisbug Jan 23, 2021
5c429a3
mannual linting round 3
zhisbug Jan 23, 2021
22f09cc
linting 4
zhisbug Jan 23, 2021
31f2315
remove legacy code
zhisbug Jan 23, 2021
65ae84f
minor
zhisbug Jan 23, 2021
2b46a1e
fix CI
zhisbug Jan 23, 2021
9dd9312
address Richard comments
zhisbug Jan 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions python/ray/util/collective/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
from ray.util.collective.collective import nccl_available, mpi_available, \
from ray.util.collective.collective import nccl_available, gloo_available, \
is_group_initialized, init_collective_group, destroy_collective_group, \
get_rank, get_world_size, allreduce, barrier, reduce, broadcast, \
allgather, reducescatter, send, recv
declare_collective_group, get_rank, get_world_size, allreduce, \
allreduce_multigpu, barrier, reduce, reduce_multigpu, broadcast, \
broadcast_multigpu, allgather, allgather_multigpu, reducescatter, \
reducescatter_multigpu, send, send_multigpu, recv, recv_multigpu

__all__ = [
"nccl_available", "mpi_available", "is_group_initialized",
"init_collective_group", "destroy_collective_group", "get_rank",
"get_world_size", "allreduce", "barrier", "reduce", "broadcast",
"allgather", "reducescatter", "send", "recv"
"nccl_available", "gloo_available", "is_group_initialized",
"init_collective_group", "destroy_collective_group",
"declare_collective_group", "get_rank", "get_world_size", "allreduce",
"allreduce_multigpu", "barrier", "reduce", "reduce_multigpu", "broadcast",
"broadcast_multigpu", "allgather", "allgather_multigpu", "reducescatter",
"reducescatter_multigpu", "send", "send_multigpu", "recv", "recv_multigpu"
]
Loading