Error in dxtbx.image_average

[phyy-nx]

From @dwpaley, after downloading a SACLA h5 file:

mpirun -n 32 dxtbx.image_average --mpi True --verbose ../data/run1/run371999-0.h5
Processing ../data/run1/run371999-0.h5: 0
Processing ../data/run1/run371999-0.h5: 32
Processing ../data/run1/run371999-0.h5: 64
Processing ../data/run1/run371999-0.h5: 96
Processing ../data/run1/run371999-0.h5: 128
Processing ../data/run1/run371999-0.h5: 160
Traceback (most recent call last):
  File "/net/viper/raid1/dwpaley/xfelgui2/w/dials-v3-0-4/build/../modules/dxtbx/command_line/image_average.py", line 435, in <module>
    sys.exit(run())
  File "/net/viper/raid1/dwpaley/xfelgui2/w/dials-v3-0-4/build/../modules/dxtbx/command_line/image_average.py", line 312, in run
    results = comm.gather(results, root=0)
  File "mpi4py/MPI/Comm.pyx", line 1262, in mpi4py.MPI.Comm.gather
  File "mpi4py/MPI/msgpickle.pxi", line 680, in mpi4py.MPI.PyMPI_gather
  File "mpi4py/MPI/msgpickle.pxi", line 685, in mpi4py.MPI.PyMPI_gather
  File "mpi4py/MPI/msgpickle.pxi", line 148, in mpi4py.MPI.Pickle.allocv
  File "mpi4py/MPI/msgpickle.pxi", line 139, in mpi4py.MPI.Pickle.alloc
SystemError: Negative size passed to PyBytes_FromStringAndSize

My testing finds it works with 10 processors but not 32. My googling indicates we are hitting a pickle limitation: https://groups.google.com/g/mpi4py/c/r95uGPcqXLA?pli=1. I think with 32 processors, we are sending 32 results back to rank 0, which apparently spills over the 2GB pickle limit.

Anyway, the proper way to do it to not use comm.gather at all and to instead use comm.reduce (like I did in my original implementation using lcls only data). Seems doable.