A new binomial scatter using packed data on intermediary processes.

[bosilca]

This PR provides an alternative solution to #8283, where all temporaries are handled as packed data resulting in less memory allocated for storing the temporary data.

Signed-off-by: George Bosilca bosilca@icl.utk.edu

[ggouaillardet]

@bwbarrett gcc5 test failed because of a full disk

ar: `u' modifier ignored since `D' is the default (see `U')
ar: .libs/libmpi_usempif08_pmpi.a: No space left on device
Makefile:2077: recipe for target 'libmpi_usempif08_pmpi.la' failed

[ggouaillardet]

@bosilca is this supposed to always work on an heterogeneous cluster (mixing big and little endian for example) ?

[mkurnosov]

@bosilca thank you. I did not find any bugs. I have tested for contiguous and non-contiguous data types, for all roots and MPI_IN_PLACE.

One proposal: it is possible to reduce memory consumption in non-root and non-leaf processes: scount * (size + 1) / 2 it is an upper bound. We can compute subtree size and allocate buffer of corresponding size:

int vparent = (bmtree->tree_prev - root + size) % size;
int subtree_size = vrank - vparent;
if (size - vrank < subtree_size)
    subtree_size = size - vrank;
packed_size = scount * subtree_size;

[bosilca]

@ggouaillardet this version is almost heterogeneous. The only remaining issue is that all participants except the root, will need to create a convertor for the root architecture (from the ompi_proc_t structure corresponding to the root) and use this convertor to unpack the local part of the message. This is true for non-leaves (where we already have an unpack operation), but also for the leaves where we will need a temporary buffer and an unpack.

[bosilca]

@mkurnosov sure, it makes sense to reduce the temporary memory for non-leaves. If we are looking at optimizations, we could also avoid packing the local part on the root, and or interleaving pack and send on the root (which will also lead to less memory required for temporaries).

[jsquyres]

bot:aws:retest

[gpaulsen]

bot:aws:retest

[jsquyres]

@bosilca Should we merge this PR? (and close #8283 and cherry pick to v4.0.x and v4.1.x, and then close #8285?)

[bosilca]

I'm done with this. I was under the impression that @mkurnosov wanted to add some optimizations, but it's entirely up to him.

[jsquyres]

I think that we should merge this PR, and if optimization comes in later, great. This fixes a real bug, so we should go ahead and merge it.

[jsquyres]

Review by proxy for @mkurnosov

[mkurnosov]

@bosilca @jsquyres I am sorry, I will prepare a small fix in the near future.