[bug]it look like a potential bug that use a local variable for asynchronous copy

[shizhixing]

in file src/channels.cc

Line:50
for (int r = 0; r < nRanks; r++) {
uintptr_t addr = (uintptr_t)(comm->sharedRes->devPeers[channelId] + comm->topParentRanks[r]);
NCCLCHECK(ncclCudaMemcpyAsync((uintptr_t*)(channel->devPeers + r), (uintptr_t*)&addr, 1, sharedRes->deviceStream.cudaStream));>>>>>>>>>>>&addr could be rewrited by next loop before doing memcpy
channel->devPeersHostPtr[r] = (struct ncclDevChannelPeer*)addr;
}
}

and also Line:87, Line 97

[sjeaugey]

Indeed, that seems like an issue. Now if it ever worked, it could mean that CUDA is actually copying the data right away (maybe for small sizes) so it ends up working.

Fixing the issue may be complicated too as we'd need to free the buffer only when the operation is complete, which may not be easy.

Thanks for the report, we'll see how to fix that.