Issue with distributing hello_world_distributed on several nodes

[dkeyGit]

Hello,

my university project group and me have some issues in relation to run the “hello_world_distributed” program. We want to run this program with SLURM on two nodes and the partition is called “qdr”. We start the running process with the following command:

srun -p qdr -N 2 hello_world_distributed

Unfortunately, we get the following error message per node:

Could not find usable bootstrap parcelport.
Preconfigured bootstrap parcelport: 'tcp'
List of available parcelports:
mpi, priority: -100, can bootstrap: 1

With the following command we get the debug information:
srun -p qdr -N 2 hello_world_distributed --hpx:list-parcel-ports --hpx:debug-clp

The output is here:
output.txt

We already searched for possible solutions and found an issue entry in #4891. There we found the tip to expand the srun-command with the following options:

srun -p qdr -N 4 hello_world_distributed --hpx:ini=hpx.parcel.mpi.enable!=1 --hpx:ini=hpx.parcel.bootstrap!=mpi

Unfortunately, now we get the error message:

terminate called without an active exception
Could not find usable bootstrap parcelport.
Preconfigured bootstrap parcelport: 'mpi'
List of available parcelports:
mpi, priority: -100, can bootstrap: 1

Here are our CmakeLists.txt and load-env.sh:
CMakeLists.txt
load-env.txt

Do you have any ideas how to fix the errors?

[hkaiser]

You will need to launch your application using mpiexec/mpirun If you want to use the mpi parcelport. A good example on how to do this can be found here: https://github.com/STEllAR-GROUP/hpx/wiki/Running-HPX-on-Rostam#running-hpx-applications-on-compute-nodes. So you should probably use something like:

salloc -p qdr -N 2 mpirun ./bin/hello_world_distriuted

[pauleonix]

"False negative", the processes were running on the headnode.

[dkeyGit]

Unfortunately it still does not work. With "salloc -p qdr -N 2 mpirun -n 2 hello_world_distributed --hpx:threads 2", we get the following output:
hello_world_distributed_sample.txt

We thought, that the program is distributed among 2 nodes (because there is "hello world" from 2 localities) on the qdr-partition, but after an analysis it turned out, that the program ran only on the headnode (not qdr-partition).

With "salloc -p qdr -N 2 mpirun -n 2 hello_world_distributed --hpx:threads 2 --hpx:debug-clp" we get the following output:
debug.txt

The command for showing the hostname ("salloc -p qdr -N 2 mpirun -n 2 hostname") only shows the headnode two times.

[pauleonix]

I now tried using sbatch and it seems to work without a problem, so this might be an issue with salloc specifically or rather with running mpirun from the headnode instead of from one of the compute nodes:

$ cat test.sh
#!/bin/sh

spack load hpx

mpirun hostname

mpirun hello_world_distributed
$ sbatch -p qdr -N 2 test.sh
Submitted batch job 36856
$ # ... wait for completion ...
$ cat slurm-36856.out
ziti-qdr008.ziti.uni-heidelberg.de
ziti-qdr009.ziti.uni-heidelberg.de
hpx::init: command line warning: --hpx:localities used when running with SLURM, requesting a different number of localities (2) than have been assigned by SLURM (1), the application might not run properly.
hello world from OS-thread 0 on locality 0
hello world from OS-thread 0 on locality 1

[hkaiser]

BTW, you should be able to add --hpx:ignore-batch-env to the command line to avoid the warning.

[dkeyGit]

On this way it also works with our own compiled version of the hello_world_distribution program.

[hkaiser]

Try:

salloc -p qdr -N 2 mpirun -n 2 ./bin/hello_world_distriuted

This all depends on how your slurm/MPI integration was built and configured. We might have to try different things.