[Core] Add SetRank method to GlobalPointer + documentation

[loumalouomega]

📝 Description

Add SetRank method to GlobalPointer, plus adding documentation

🆕 Changelog

• Add SetRank method to GlobalPointer + documentation

[roigcarlo]

Why exactly do you need to change the rank of a global pointer?

GP were designed to be created in the partition they belong, its hard for me to find a use case in which this construction is valid and does not lead to problems.

[loumalouomega]

Why exactly do you need to change the rank of a global pointer?

GP were designed to be created in the partition they belong, its hard for me to find a use case in which this construction is valid and does not lead to problems.

Because somethimes the rank is not provided in construction time

[roigcarlo]

Well yes, but can you elaborate? how can you not have the rank in the processor its mean to create the GP?

[loumalouomega]

Well yes, but can you elaborate? how can you not have the rank in the processor its mean to create the GP?

Because the GeometricalObjectBins generates SpatialSearchResult, which contains a GlobalPointer, but does not provide the rank, so it is assumed to be 0 always.

See: d6873a9

[roigcarlo]

if this is the problem I would personally change the global_pointer.h default to something like:

GlobalPointer(TDataType data, int rank = ParallelEnvironment::GetDefaultRank())

But maybe I am biased here, ping @KratosMultiphysics/mpi

[loumalouomega]

if this is the problem I would personally change the global_pointer.h default to something like:

GlobalPointer(TDataType data, int rank = ParallelEnvironment::GetDefaultRank())

But maybe I am biased here, ping @KratosMultiphysics/mpi

That will not work, because creates a MPI dependency, will fail on linking time

[roigcarlo]

It does not work like that

When you start Kratos with mpi, the first time the kernel is loaded you will call:

# __init__.py:67
mpi.InitializeMPIParallelRun()

This code lives in the MPI extension of the core, and among other things does:

// Define the World DataCommunicator as a wrapper for MPI_COMM_WORLD and make it the default.
ParallelEnvironment::RegisterDataCommunicator("World", MPIDataCommunicator::Create(MPI_COMM_WORLD), ParallelEnvironment::MakeDefault);

This will invoke the code from:

void ParallelEnvironment::SetAsDefault(
    std::unordered_map<std::string, DataCommunicator::UniquePointer>::iterator& rThisCommunicator)
{
    mDefaultCommunicator = rThisCommunicator;
    const auto& r_comm = *(rThisCommunicator->second);
    mDefaultRank = r_comm.Rank();
    mDefaultSize = r_comm.Size();
}

Which will set the mDefaultcommunicator, mDefaultRank and mDefaultSize be the ones from the communicator you just created.

It does not matter that the implementation this part is MPI dependent, because it does not live in the same shared lib as the rest of KratosCore, but it will change the default registered communicator.

When you call ParallelEnvironment::GetDefaultRank(), (which does not depend on mpi) you will access proper process rank, size, or mpi communicator, even if they are mpi dependent.

[roigcarlo]

Btw, I will admit that SetRank would be usefull in one particular situation, which is trying to transfer GP which belong to more than one communicator (Kratos default one + other custom registered comm) with different rank ids. And even in those i am not sure if its better to create another copy of the pointer when in the context of the correct communicator.

[loumalouomega]

It does not work like that

When you start Kratos with mpi, the first time the kernel is loaded you will call:

# __init__.py:67
mpi.InitializeMPIParallelRun()

This code lives in the MPI extension of the core, and among other things does:

// Define the World DataCommunicator as a wrapper for MPI_COMM_WORLD and make it the default.
ParallelEnvironment::RegisterDataCommunicator("World", MPIDataCommunicator::Create(MPI_COMM_WORLD), ParallelEnvironment::MakeDefault);

This will invoke the code from:

void ParallelEnvironment::SetAsDefault(
    std::unordered_map<std::string, DataCommunicator::UniquePointer>::iterator& rThisCommunicator)
{
    mDefaultCommunicator = rThisCommunicator;
    const auto& r_comm = *(rThisCommunicator->second);
    mDefaultRank = r_comm.Rank();
    mDefaultSize = r_comm.Size();
}

Which will set the mDefaultcommunicator, mDefaultRank and mDefaultSize be the ones from the communicator you just created.

It does not matter that the implementation this part is MPI dependent, because it does not live in the same shared lib as the rest of KratosCore, but it will change the default registered communicator.

When you call ParallelEnvironment::GetDefaultRank(), (which does not depend on mpi) you will access proper process rank, size, or mpi communicator, even if they are mpi dependent.

I will try

[roigcarlo]

Well let see what the rest of people say, maybe I am being paranoid and setrank is fine for everyone...

[loumalouomega]

It does not work like that
When you start Kratos with mpi, the first time the kernel is loaded you will call:

# __init__.py:67
mpi.InitializeMPIParallelRun()

This code lives in the MPI extension of the core, and among other things does:

// Define the World DataCommunicator as a wrapper for MPI_COMM_WORLD and make it the default.
ParallelEnvironment::RegisterDataCommunicator("World", MPIDataCommunicator::Create(MPI_COMM_WORLD), ParallelEnvironment::MakeDefault);

This will invoke the code from:

void ParallelEnvironment::SetAsDefault(
    std::unordered_map<std::string, DataCommunicator::UniquePointer>::iterator& rThisCommunicator)
{
    mDefaultCommunicator = rThisCommunicator;
    const auto& r_comm = *(rThisCommunicator->second);
    mDefaultRank = r_comm.Rank();
    mDefaultSize = r_comm.Size();
}

Which will set the mDefaultcommunicator, mDefaultRank and mDefaultSize be the ones from the communicator you just created.
It does not matter that the implementation this part is MPI dependent, because it does not live in the same shared lib as the rest of KratosCore, but it will change the default registered communicator.
When you call ParallelEnvironment::GetDefaultRank(), (which does not depend on mpi) you will access proper process rank, size, or mpi communicator, even if they are mpi dependent.

I will try

It is impossible, I need many forward declarations in many different classes (20 files I have right now). It is easier to just simply keet the SetRank method

[roigcarlo]

Actually, the problem is the FillCommunicator. Let me take a look

[loumalouomega]

Actually, the problem is the FillCommunicator. Let me take a look

yes, but I needed to change many files

[roigcarlo]

Ok to many things are wrong in the core. Lets accept this for now and I will take care of unraveling the kernel

[loumalouomega]

Ok to many things are wrong in the core. Lets accept this for now and I will take care of unraveling the kernel

Indeed..., many many cross dependencies. I think we should reconsider the forward declaration policy to begin with

[RiccardoRossi]

why do you want to initialize it? if it is not a correct value better leave it undefined no?

[loumalouomega]

The diff is terrible, the code was already like that , see line 48 in previous implementation

[RiccardoRossi]

well, i guess we should have a debug error here in case we are not in mpi mode

[loumalouomega]

And rank is different than 0, because if you implement a generic code in serial it will set always to 0, otherwise you will create many false errors

[RiccardoRossi]

well, imagine you try to set something to rank 5 in serial... that would be silently ignored

[RiccardoRossi]

Ok to many things are wrong in the core. Lets accept this for now and I will take care of unraveling the kernel

Indeed..., many many cross dependencies. I think we should reconsider the forward declaration policy to begin with

i am fundamentally against it!

[loumalouomega]

Ok to many things are wrong in the core. Lets accept this for now and I will take care of unraveling the kernel

Indeed..., many many cross dependencies. I think we should reconsider the forward declaration policy to begin with

i am fundamentally against it!

In that case we will have longer compilation times and cross dependencies issues all along the core

[RiccardoRossi]

well, we will need to solve those... forward declaration is really a dangerous thing ...

[loumalouomega]

well, we will need to solve those... forward declaration is really a dangerous thing ...

When it is dangerous?, if a same class is defined diferently in different files?, a priori you will have linking problems

[RiccardoRossi]

for example the dependency check mechanism (includes) gets broken by forward declarations

[loumalouomega]

for example the dependency check mechanism (includes) gets broken by forward declarations

I think if you what are you doing should not be problematic, and we approve changes little by little

[RiccardoRossi]

well, i ping @KratosMultiphysics/technical-committee ... but i anticpate you that i am completely agains this

[loumalouomega]

well, i ping @KratosMultiphysics/technical-committee ... but i anticpate you that i am completely agains this

I understand, but @roigcarlo also found due to this PR problems related with cross dependencies difficult to solve without this technique