NCCL inplace reduce-scatterv trashes rank 0 buffer

[ndryden]

We currently implement the reduce-scatter as a reduce to rank 0 followed by a scatterv. When doing an in-place op, the reduce is in-place on the input sendbuf. This therefore writes to portions of sendbuf on rank 0 that are outside of the region where the final scattered value would be placed.

I don't find something explicitly prohibiting this in the MPI standard, but:

• It's a bit aesthetically displeasing.
• In other cases, like a MPI_Recv with a buffer/count larger than the actual message length, MPI does guarantee that no more memory will be touched than is actually needed by the message.
• Avoiding it shouldn't take too much overhead if we use a memory pool.
• A better, direct implementation can probably avoid it.