ompi5.0.0rc10 - mapping issues - PMIX

[thomasgillis]

Background information

I have PMIX errors when I request non-straightforward mpiexec bindings.
I have 2 nodes of 128 cores each, with 2 MPI processes and 16 threads I am trying to get:

node 0: core:[0-15]
node 1: core:[0-15]

To achieve this I use

mpiexec -np 2 --map-by ppr:1:node:pe=16

But then I get some errors from PMIX:

PMIX ERROR: OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750

I am not 100% confident on the --map-by ppr:1:node:pe=16 command but the segfault from pmix seems suspicious as well.
Is my command correct? is there something I need to change to get rid of the pmix error?

details:

• ompi-5.0.0rc10 built from source

[hppritcha]

there's a remote chance this may be related to use of cma.
Could you rerun your test with

--mca smsc none

and see if you see the PMIX error?

[hppritcha]

so did PMIx still emit the error message with CMA disabled?

[thomasgillis]

here is what I get:

==================================================================
 mpi cmd:  /home/users/u100155/lib-OMPI-5.0.0rc10-UCX-1.13.1/bin/mpiexec -np 2 --map-by ppr:1:node:pe=16 --mca smsc none
------------------------------------------------------------------
--------------------------------------------------------------------------
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:      mel0429
Framework: smsc
Component: none
--------------------------------------------------------------------------
^@--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  mca_base_framework_open on opal_smsc failed
  --> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
^@*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[mel0429:18037] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
prterun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [prterun-mel0429-18023@1,0]
  Exit code:    14
--------------------------------------------------------------------------

apparently I don't have smsc, is that expected?

[hppritcha]

okay i got syntax wrong, try

--mca smsc ^cma

[hppritcha]

smsc is an optimization for intra-node long message transfers but is not essential for correct operation.

[thomasgillis]

the PMIX error is still there:

 mpi cmd:  /home/users/u100155/lib-OMPI-5.0.0rc10-UCX-1.13.1/bin/mpiexec -np 2 --map-by ppr:1:node:pe=16 --mca smsc ^cma
------------------------------------------------------------------
[mel0429:29695] PMIX ERROR: OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750

[thomasgillis]

@hppritcha FYI when the binding leads to an intra-node layout the error is not there

 mpi cmd:  /home/users/u100155/lib-OMPI-5.0.0rc10-UCX-1.13.1/bin/mpiexec -np 2 --map-by ppr:1:node:pe=16 --report-bindings --hostfile /home/users/u100155/temp_hostfile_2_16.txt
------------------------------------------------------------------
[mel0429:06002] Rank 0 bound to package[0][core:0-15]
[mel0430:03768] Rank 1 bound to package[0][core:0-15]
[mel0429:06002] PMIX ERROR: OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750

vs

==================================================================
 mpi cmd:  /home/users/u100155/lib-OMPI-5.0.0rc10-UCX-1.13.1/bin/mpiexec -np 2 --cpus-per-rank 16 --report-bindings --hostfile nodefile
------------------------------------------------------------------
[mel0429:13751] Rank 0 bound to package[0][core:0-15]
[mel0429:13751] Rank 1 bound to package[0][core:16-31]

[rhc54]

The command is correct and leads to the correct mapping, at least on PRRTE master:

$ prterun --prtemca ras_simulator_num_nodes 2 --prtemca hwloc_use_topo_file /Users/rhc/pmix/topologies/summit.h17n08.lstopo-2.2.0.xml --map-by ppr:1:node:pe=16 --display map-devel hostname

=================================   JOB MAP   =================================
Data for JOB prterun-Ralphs-iMac-2-22235@1 offset 0 Total slots allocated 84
Mapper requested: NULL  Last mapper: ppr  Mapping policy: BYNODE:NOOVERSUBSCRIBE  Ranking policy: SLOT
Binding policy: HWTHREAD:IF-SUPPORTED  Cpu set: N/A  PPR: 1:node  Cpus-per-rank: 16  Cpu Type: HWT
Num new daemons: 0	New daemon starting vpid INVALID
Num nodes: 2

Data for node: nodeA0	State: 3	Flags: MAPPED:SLOTS_GIVEN
        Daemon: [prterun-Ralphs-iMac-2-22235@0,1]	Daemon launched: False
            Num slots: 42	Slots in use: 1	Oversubscribed: FALSE
            Num slots allocated: 42	Max slots: 42	Num procs: 1
        Data for proc: [prterun-Ralphs-iMac-2-22235@1,0]
                Pid: 0	Local rank: 0	Node rank: 0	App rank: 0
                State: INITIALIZED	App_context: 0
        	Binding: package[0][hwt:0-15]

Data for node: nodeA1	State: 3	Flags: MAPPED:SLOTS_GIVEN
        Daemon: [prterun-Ralphs-iMac-2-22235@0,2]	Daemon launched: False
            Num slots: 42	Slots in use: 1	Oversubscribed: FALSE
            Num slots allocated: 42	Max slots: 42	Num procs: 1
        Data for proc: [prterun-Ralphs-iMac-2-22235@1,1]
                Pid: 0	Local rank: 0	Node rank: 0	App rank: 1
                State: INITIALIZED	App_context: 0
        	Binding: package[0][hwt:0-15]

Warning: This map has been generated with the DONOTLAUNCH option;
	The compute node architecture has not been probed, and the displayed
	map reflects the HEADNODE ARCHITECTURE. On systems with a different
	architecture between headnode and compute nodes, the map can be
	displayed using `prte --display map /bin/true`, which will launch
	enough of the DVM to probe the compute node architecture.

=============================================================

I used the topology from Summit as it matches the one described.

[thomasgillis]

@rhc54 ok, thanks for the confirmation.
Any idea on the error from pmix?
The system I use is IB (so I have built ucx-1.13.1) :-)

EDIT: FYI I have built rc9 and everything works fine, so I guess it has been introduced recently

==================================================================
 mpi cmd:  /home/users/u100155/lib-OMPI-5.0.0rc9-UCX-1.13.1/bin/mpiexec -np 2 --map-by ppr:1:node:pe=16 --report-bindings
------------------------------------------------------------------
[mel0385:19899] Rank 0 bound to package[0][core:0-15]
[mel0386:30153] Rank 1 bound to package[0][core:0-15]

[rhc54]

No ideas, I'm afraid. Looks like it is coming from an application process? If so, then I think I've seen some OMPI bug reports about incorrect data retrieval for IB transports - not sure if anyone has addressed those.

[hppritcha]

is there anything special about the application your are trying to launch? I'd like to be able to reproduce.

[thomasgillis]

No it's a ping-pong/osu bandwidth measurement.
I use thread multiple but I might not be relevant to the issue