Repository navigation
[GRAPH] Use channel shuffling only for IB systems - #1228
Merged
Merged
Conversation
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 26, 2024 07:47
f5725ca to
8ee34be
Compare
gilbertlee-amd
approved these changes
Jun 26, 2024
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 26, 2024 15:27
8ee34be to
440b671
Compare
gilbertlee-amd
approved these changes
Jun 26, 2024
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 26, 2024 16:30
440b671 to
a3ce2b6
Compare
nileshnegi
marked this pull request as ready for review
June 26, 2024 16:31
nileshnegi
requested review from
AtlantaPepsi,
KawtharShafie,
PedramAlizadeh,
akolliasAMD,
corey-derochie-amd,
edgargabriel,
haripriyaayyalasomayajula,
mberenjk,
nusislam and
wenkaidu
as code owners
June 26, 2024 16:31
wenkaidu
approved these changes
Jun 26, 2024
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 26, 2024 16:38
a3ce2b6 to
6c5c4f2
Compare
wenkaidu
reviewed
Jun 26, 2024
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 27, 2024 16:15
6c5c4f2 to
6eddaa7
Compare
wenkaidu
approved these changes
Jun 27, 2024
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 28, 2024 14:22
6eddaa7 to
0ae3838
Compare
wenkaidu
reviewed
Jun 28, 2024
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
nileshnegi
force-pushed
the
fix/gid-channel-shuffling
branch
from
June 28, 2024 17:01
f77670c to
120f93f
Compare
wenkaidu
approved these changes
Jun 28, 2024
Signed-off-by: nileshnegi <Nilesh.Negi@amd.com>
wenkaidu
approved these changes
Jul 2, 2024
nileshnegi
added a commit
to nileshnegi/rccl
that referenced
this pull request
Jul 2, 2024
* [GRAPH] Use channel shuffling only for IB systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> * [GRAPH] Define channels=48 for gfx94 RoCE systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> * [GRAPH] Increase channels for RoCE gfx94 systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> --------- Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> (cherry picked from commit 5be3b71)
akolliasAMD
pushed a commit
to akolliasAMD/rccl
that referenced
this pull request
Jul 3, 2024
* [GRAPH] Use channel shuffling only for IB systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> * [GRAPH] Define channels=48 for gfx94 RoCE systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> * [GRAPH] Increase channels for RoCE gfx94 systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> --------- Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> (cherry picked from commit 5be3b71)
lajagapp
pushed a commit
to lajagapp/rccl
that referenced
this pull request
Jul 4, 2024
* [GRAPH] Use channel shuffling only for IB systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> * [GRAPH] Define channels=48 for gfx94 RoCE systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> * [GRAPH] Increase channels for RoCE gfx94 systems Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> --------- Signed-off-by: nileshnegi <Nilesh.Negi@amd.com> (cherry picked from commit 5be3b71)
nileshnegi
added a commit
to nileshnegi/rccl
that referenced
this pull request
Jul 15, 2024
This reverts commit 6ae3c0e.
nileshnegi
added a commit
to nileshnegi/rccl
that referenced
this pull request
Jul 15, 2024
This reverts commit 5be3b71.
nileshnegi
added a commit
to nileshnegi/rccl
that referenced
this pull request
Jul 15, 2024
This reverts commit 5be3b71.
nileshnegi
added a commit
that referenced
this pull request
Jul 18, 2024
1 task
nileshnegi
added a commit
to nileshnegi/rccl
that referenced
this pull request
Jul 18, 2024
This reverts commit 6ae3c0e.
1 task
BertanDogancay
pushed a commit
that referenced
this pull request
Jan 23, 2025
Rework core for NVIDIA Trusted Computing * Compress work structs so that they are shared between channels * Utilize the full amount of kernel argument space permitted (4k) before resorting to work fifo. * Rework the task preprocessing phase. * Use a separate abortDevFlag which is kept in sync with abortFlag using cudaMemcpy operations. * Rename src/include/align.h to src/include/bitops.h Add lazy connection establishment for collective operations * Move buffer allocation and connection establishment to the first collective operation using that algorithm. * Accelerate init time and reduce memory usage. * Avoid allocating NVLS buffers if all calls are registered. * Compute algo/proto in ncclLaunchCollTasksInfo early on. * Connect peers in ncclCollPreconnectFunc if not connected already. * Also move shared buffer creation to the first send/recv call. Accelerate intra-node NVLink detection * Make each rank only detect NVLinks attached to its GPU. * Fuse XMLs to reconstruct the full NVLink topology Add init profiling to report time spend in different init phases. * Report timings of bootstrap, allgather, search, connect, etc. * Add new "PROFILE" category for NCCL_DEBUG_SUBSYS. Add support for PCI p2p on split PCI switches * Detect split PCI switches through a kernel module exposing switch information. * Update the topology XML and graph to add those inter-switch connections. Add cost estimation API * Add a new ncclGroupEndSimulate primitive to return the estimated time a group would take. Net/IB: Add separate traffic class for fifo messages * Add NCCL_IB_FIFO_TC to control the traffic class of fifo messages independently from NCCL_IB_TC. Merges PR #1194 Net/IB: Add support for IB router * Use flid instead of lid if subnets do not match * Warn if flid is 0 Optimizations and fixes for device network offload (unpack) * Double the default number of channels * Cache netDeviceType * Fix save/increment head logic to enable Tree support. Support ncclGroupStart/End for ncclCommAbort/Destroy * Allow Abort/Destroy to be called within a group when managing multiple GPUs with a single process. Improve Tuner API * Provide to the plugin the original cost table so that the plugin can leave unknown or disabled algo/proto combinations untouched. * Remove nvlsSupport and collnetSupport. Do not print version to stdout when using a debug file * Also print version from all processes with INFO debug level. Fixes issue #1271 Fix clang warnings in NVTX headers * Update NVTX headers to the latest version Fixes issue #1270 Disable port fusion in heterogeneous systems * Do not fuse ports if a mix of multi-port and single port are detected. Fix NVLS graphs search for dual NICs. * Fix NVLS graph search when we have more than one NIC per GPU. Fix crash with collnetDirect * Add separate graph search for collnetDirect, testing alltoall paths and working similarly to the NVLS search. Fix hang when nodes have different CPU types * Add the CPU type to the rank peer info. * Align all ranks on the CPU type after the first allgather. * Only use the aligned CPU type for all tuning operations. Fixes issue #1136 Fixes issue #1184 Fix performance of registered send/recv operations * Allow for single full size operations * Add INFO to confirm the registration of send/recv buffers. Move all sync ops to finalize stage * Ensure ncclCommDestroy is non-blocking if ncclCommFinalize has been called. Improve error reporting during SHM segment creation Improve support of various compilers Merges PR #1177 Merges PR #1228 Allow net and tuner plugins to be statically linked * Search for ncclNet or ncclTuner symbols in the main binary. Merges PR #979 Plugin examples includes cleanup * Harmonize err.h and common.h usage. * Add mixed plugin with both net and tuner.
BertanDogancay
pushed a commit
that referenced
this pull request
Mar 28, 2025
Add scalable init API * Add new ncclCommInitRankScalable to allow for passing multiple unique IDs to the init function. * Spreads the load onto multiple bootstrap roots, allowing for constant bootstrap time. * Requires multiple ranks to create a unique ID, and the CPU-side ID exchange code to call allgather[v] instead of broadcast. Accelerate init bootstrap operations * Reduce the number of calls to allgather. * Allow roots to reply early to ranks when information is already available. * Add an option to use ncclNet instead of sockets to perform bootstrap allgather operations. Add PAT algorithms for Allgather and ReduceScatter * Parallel Aggregated Trees, variation of Bruck algorithm. * Logarithmic number of network steps for small sizes at scale. * Only supports one rank per node at the moment. Add support for registered buffers for intra-node communication. * Allow registered user buffers to be accessed directly intra-node * Avoids extra copies in algorithms which permit it, saving memory bandwidth and helping with compute overlap. Add profiler plugin API * New plugin API for profiling * Supports various levels of profiling, with a hierarchy. Asynchronous graph allocation * Make calls to cudaMalloc and cudaMemcpy during graph allocation asynchronous. * Significantly speeds up graph capture. Use fatal IB asynchronous events to stop network operation * Avoids many other error messages * Only fatal errors are affected; potentially transient errors (e.g. port down) do not cause an immediate stop. Set P2P level to PXB on AMD CPUs when using more than 2 GPUs per node * P2P would cause a significant performance degradation when using many GPUs, and therefore many interleaved data flows. * Disable P2P through the CPU when we have 3+ GPUs per node; keep it enabled when we only have 2 GPUs. Improve the init logs to report the real NCCL function. * Make the log report ncclCommInitRank or ncclCommSplit, rather than the generic ncclCommInitRankFunc. Add a parameter to set the location of the user configuration file. * Add NCCL_CONF_FILE environment variable to set where the user's configuration file resides. Increase default IB timeout * Increase IB timeout value from 18 to 20. * Should help avoid fatal errors on large RoCE systems. Add new check for nvidia peermem * On linux kernels 6.6+, /sys/kernel/mm/memory_peers is no longer present; check for /sys/module/nvidia_peermem/version instead. Fix old performance regression when mixing small and large operations. * Improves distribution of work on channels. Fix crash when NUMA IDs are equal to -1. * Can happen when a NIC is a virtual NIC, or when linux doesn't know which NUMA node a device is attached to * Issue NVIDIA/nccl-tests#233 Fix tree graph search when NCCL_CROSS_NIC is set to 1. * Would force NCCL to use the balanced_tree pattern, thereby disabling LL128 on platforms with 1 GPU+1 NIC per PCI switch. * Would also try to use alternate rings even though it was not needed. Compiler tweaks and fixes * PR #1177 * PR #1228 Fix stack smash * PR #1325 Fixes for multi-node NVLink + IB operation Coverity fixes and comments.
corey-derochie-amd
pushed a commit
to corey-derochie-amd/rccl
that referenced
this pull request
Apr 9, 2025
Add scalable init API * Add new ncclCommInitRankScalable to allow for passing multiple unique IDs to the init function. * Spreads the load onto multiple bootstrap roots, allowing for constant bootstrap time. * Requires multiple ranks to create a unique ID, and the CPU-side ID exchange code to call allgather[v] instead of broadcast. Accelerate init bootstrap operations * Reduce the number of calls to allgather. * Allow roots to reply early to ranks when information is already available. * Add an option to use ncclNet instead of sockets to perform bootstrap allgather operations. Add PAT algorithms for Allgather and ReduceScatter * Parallel Aggregated Trees, variation of Bruck algorithm. * Logarithmic number of network steps for small sizes at scale. * Only supports one rank per node at the moment. Add support for registered buffers for intra-node communication. * Allow registered user buffers to be accessed directly intra-node * Avoids extra copies in algorithms which permit it, saving memory bandwidth and helping with compute overlap. Add profiler plugin API * New plugin API for profiling * Supports various levels of profiling, with a hierarchy. Asynchronous graph allocation * Make calls to cudaMalloc and cudaMemcpy during graph allocation asynchronous. * Significantly speeds up graph capture. Use fatal IB asynchronous events to stop network operation * Avoids many other error messages * Only fatal errors are affected; potentially transient errors (e.g. port down) do not cause an immediate stop. Set P2P level to PXB on AMD CPUs when using more than 2 GPUs per node * P2P would cause a significant performance degradation when using many GPUs, and therefore many interleaved data flows. * Disable P2P through the CPU when we have 3+ GPUs per node; keep it enabled when we only have 2 GPUs. Improve the init logs to report the real NCCL function. * Make the log report ncclCommInitRank or ncclCommSplit, rather than the generic ncclCommInitRankFunc. Add a parameter to set the location of the user configuration file. * Add NCCL_CONF_FILE environment variable to set where the user's configuration file resides. Increase default IB timeout * Increase IB timeout value from 18 to 20. * Should help avoid fatal errors on large RoCE systems. Add new check for nvidia peermem * On linux kernels 6.6+, /sys/kernel/mm/memory_peers is no longer present; check for /sys/module/nvidia_peermem/version instead. Fix old performance regression when mixing small and large operations. * Improves distribution of work on channels. Fix crash when NUMA IDs are equal to -1. * Can happen when a NIC is a virtual NIC, or when linux doesn't know which NUMA node a device is attached to * Issue NVIDIA/nccl-tests#233 Fix tree graph search when NCCL_CROSS_NIC is set to 1. * Would force NCCL to use the balanced_tree pattern, thereby disabling LL128 on platforms with 1 GPU+1 NIC per PCI switch. * Would also try to use alternate rings even though it was not needed. Compiler tweaks and fixes * PR ROCm#1177 * PR ROCm#1228 Fix stack smash * PR ROCm#1325 Fixes for multi-node NVLink + IB operation Coverity fixes and comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details
Work item:
Internal
What were the changes?
Why were the changes made?
Channel shuffling improves performance on IB systems, but adversely affects RoCE systems.
How was the outcome achieved?
NCCL_IB_GID_INDEXneeds to be defined as 3 for RoCE systems. If this is the case, do not use channel shuffling rings and use channels=48, else, use channel shuffling and channels=64.Approval Checklist
Do not approve until these items are satisfied.