UCX CQ failure report during initialization (Power)

[shamisp]

The following error reported during UCX initialization on power.

--------------------------------------------------------------------------
Failed to create a completion queue (CQ):

Hostname: XXXX
Requested CQE: 16384
Error:    Cannot allocate memory

Check the CQE attribute.
--------------------------------------------------------------------------

[yosefe]

This does not seem like UCX error message. Where is this coming from?

[shamisp]

I would guess - verbs driver ?

[yosefe]

no

[shamisp]

hmmm…

[shamisp]

Ok, this is a bug in OpenMPI.
They set CQE size without query and as a result it may not work on a certain platforms:
https://github.com/open-mpi/ompi/blob/master/orte/mca/oob/ud/oob_ud_qp.c#L53

[shamisp]

I reported the issue to OMPI:
open-mpi/ompi#1301