Adding Persistent Collective Communication

[tonyskjellum]

This ticket proposes non-blocking persistent collective operations be added to the standard.

For each API that currently supports a request as an out argument of a non-blocking collective operation, MPI will support an _init version.

Example: MPI_Bcast() [blocking, MPI-1] -> MPI_Ibcast(...,info,request[,ierr]) [MPI 3.x] -> MPI_Bcast_init(...,info,request[,ierr]) [new]

The attached text is prototypical standards draft text.

Some quick notes:

MPI persistent collective operation _inits are initialization calls and follow all the ordering rules already existing for posing nonblocking collective operations (this implies _init operations are local and non-blocking). Remember that the point-to-point init functions disallow communication whereas the persistent collective inits may communicate but must not rely on state of other MPI processes.

The info argument is an in argument that allows for a limited number of hints about how the persistent, nonblocking collective will be used, as defined in the text (collective chapter).

The request argument is an out argument in each process of the calling group of the communicator that can be used zero or more times to start a collective operation. Each such request must be started in all the processes of the underlying group of the communicator (they are deemed active after this). Each request must be completed (making them inactive) before another start is permitted. Each process in the group of the communicator must complete the operation.

We do not require that starts across a group for persistent collective operations serialize to the same order. This makes using MPI_STARTALL allowed.

MPI_Test/MPI_Wait operations (all) give completion information without destroying the persistent request. MPI_Request_free() is used as with persistent point-to-point operations to free up an inactive persistent collective operation. This is the same behavior as for persistent point-to-point requests.

There are some per-communicator info items also defined that make sense on a communicator scope vs. a per-operation scope, also defined in the collective chapter.

Ticket #83 specifies additional functionality for info keys for these persistent operations, and is a related ticket that should be considered in sequence or in parallel with this ticket.

The associated PR is at: https://github.com/mpi-forum/mpi-standard/pull/29

Historical note; you can see the original proposals in the MPI-2 JOD, circa 1997-98: https://www.mpi-forum.org/docs/mpi-jd/mpi-20-jod.ps .

[tonyskjellum]

topol.PDF
coll.PDF

[tonyskjellum]

These are the latest versions of the Topology and Collective chapters impacted by our proposal.

[dholmes-epcc-ed-ac-uk]

Latest version of pdf:
mpi-report.pdf

[wesbland]

I had a few concerns that I raised on the PR. None of them were significant, but it would be good to take care of them before next week.

[tonyskjellum]

OK, I''ll go look...

On Thu, Jun 2, 2016 at 8:12 AM, Wesley Bland [email protected]
wrote:

I had a few concerns that I raised on the PR. None of them were
significant, but it would be good to take care of them before next week.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA38iY10TUUwgsySG3G4RqNoXxb69s_7ks5qHtakgaJpZM4GxSUF
.

Anthony Skjellum, PhD
[email protected]
Cell: +1-205-807-4968

[schulzm]

I added a few more comments, especially on the intro section.

[hritzdorf]

Page 215, Line 5-9 of mpi-report.pdf specifies:
After initialization, all associated send buffers and buffers associated with input arguments (such as arrays of counts, displacements, or datatypes in the vector versions of the collectives) should not be modified, and all associated receive buffers should not be accessed, until the corresponding persistent request is freed.

I think, that the send buffers can be modified until MPI_Start/MPI_Startall and receive buffers can be accessed after the corresponding request is completed not freed.

[tonyskjellum]

We do not allow these operations in Startall, but you are correct:

I would agree with this modified statement: I think, that the send buffers
can be modified until MPI_Start/ and receive buffers can be accessed after
the corresponding request is completed not freed.

Tony

On Fri, Jun 3, 2016 at 5:15 AM, Hubert Ritzdorf [email protected]
wrote:

Page 215, Line 5-9 of mpi-report.pdf specifies:
After initialization, all associated send buffers and buffers associated
with input arguments (such as arrays of counts, displacements, or datatypes
in the vector versions of the collectives) should not be modified, and all
associated receive buffers should not be accessed, until the corresponding
persistent request is freed.

I think, that the send buffers can be modified until
MPI_Start/MPI_Startall and receive buffers can be accessed after the
corresponding request is completed not freed.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA38iTBIG5ZU5xmoLVDkbhwfjWEuFOf5ks5qH_7QgaJpZM4GxSUF
.

Anthony Skjellum, PhD
[email protected]
Cell: +1-205-807-4968

[schulzm]

I had the same comment - see comment on the PR - I think we have to allow this, otherwise we can only send the same data over and over again.

To be more precise, we can modify send buffers until Start and then again after the request is complete until the next Start. Similar for receive buffers.

@tonyskjellum: Can you clarify the statement that Startall is not allowed? I didn't get this from the text.

[tonyskjellum]

I will check with Dan about start all ..
At one point we discussed a restriction

The wording about changing buffers is inadvertently restrictive --- we need to be able to change buffers when the requests are inactive ...

[dholmes-epcc-ed-ac-uk]

As far as I recall, the “no buffer change” restriction is about preventing the user from changing the address of the buffer, not the content of the buffer.

[tonyskjellum]

That's logical and what I remember too now that you mention

[schulzm]

The user can't do that anyway - there is no way to access or change the pointer after the init calls - it also different to the terminology that we use in the rest of the standard (as far as I remember): changing a buffer means changing its contents (and that's a real restriction between start and completions and should be mentioned)

[RolfRabenseifner]

Hi Tony, Ryan, and Dan,

As we saw, the wording in the PT2PT Persistent Communication
chapter uses always "should" instead of "have to"/"must".

The meaning of "should" may be taken out of

http://www.iso.org/iso/how-to-write-standards.pdf

"In all clauses, you should be clear about
what is a requirement and what is a
recommendation or other statement.
ISO uses the following words to make
the distinction :
• Requirements : shall, shall not
• Recommendations : should, should not
• Permission : may, need not
• Possibility and capability : can, cannot"

Best regards
Rolf

Dr. Rolf Rabenseifner . . . . . . . . . .. email [email protected]
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

[wgropp]

I’ve added this information and some related info to the instructions document (instr.tex), which I’ll push later today.

Bill

William Gropp
Acting Director and Chief Scientist, NCSA
Director, Parallel Computing Institute
Thomas M. Siebel Chair in Computer Science
University of Illinois Urbana-Champaign

[tonyskjellum]

The associated pull request is now: https://github.com/mpi-forum/mpi-standard/pull/29

[wesbland]

@tonyskjellum / @dholmes-epcc-ed-ac-uk - Were there any "no no" changes made here that need to be read? Nothing is on the schedule, but I thought I remembered that there were a few needed after the reading (and the above comment seems to indicate that).

[tonyskjellum]

Rolf, thank you ... As we’ve explained we still intended to do more with subsequent ticket 90 and beyond if needed; maybe this recommendation could be a small enough change to fit within a ticket 25 no no vote or such ... but maybe we should do this on ticket 90 since it’s purpose is precisely to clean the front matter language for ticket 25... Anthony Skjellum, PhD 205-807-4968

…

On Feb 28, 2018, at 3:55 AM, Rolf Rabenseifner ***@***.***> wrote: In the intro of Section 3.9 on page 73 (pdf page 105), the sentence "The remainder of this section covers only point-to-point persistent operations." is not fully true. It wanted to tell that the collective persistent ...INIT... routines are not here. A more correct statement would be: "The remainder of this section covers the point-to-point persistent initialization operations and the start routines, which are used for both point-to-point and collective persistent communication." Best regards Rolf ----- Original Message ----- > From: "github notifications" ***@***.***> > To: "mpi-forum" ***@***.***> > Cc: "Rolf Rabenseifner" ***@***.***>, "Mention" ***@***.***> > Sent: Tuesday, February 27, 2018 5:58:54 PM > Subject: Re: [mpi-forum/mpi-issues] Adding Persistent Collective Communication (#25) > Here is the diff against the latest MPI-3 standard; for reading on 2/28/18. > > https://uab.box.com/s/5n6wcks2w30vb62uanfnqa6rj4txyho4 > > > > -- > You are receiving this because you were mentioned. > Reply to this email directly or view it on GitHub: > #25 (comment) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email ***@***.*** . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[tonyskjellum]

mpi32-report-ticket25-austin-vote-june2018.pdf

This is the public copy.

[schulzm]

Passed first vote during Austin Forum Meeting in June 2018

[tonyskjellum]

This passed the first vote in Austin; we will have the second vote in Barcelona.

[puribangalore]

Here is the PDF file with minor edits to fix the API for second vote in Barcelona.
mpi32-report-ticket25-barcelona-vote-sep2018.pdf

[tonyskjellum]

A request for no-no-votes and for the formal second vote were made before the two-week deadline just now.

[schulzm]

Passed no/no vote and second vote at Barcelona meeting in Sep. 2018, ready to be merged into golden copy

[tonyskjellum]

Ok ticket #80 and #105 need to update to reflect the new APIs once merger done Anthony Skjellum, PhD 205-807-4968

…

On Sep 21, 2018, at 9:16 AM, Martin Schulz ***@***.***> wrote: Passed no/no vote and second vote at Barcelona meeting in Sep. 2018, ready to be merged into golden copy — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tonyskjellum]

It would now be apropos to discuss if there are any other places in the MPI standard besides Chapters 5 and 7 where persistent collective operations should be posed... we should advance such tickets (e.g., MPI I/O).

[schulzm]

Vote tally: 16 yes, 0 abstain, 0 no - full results at https://www.mpi-forum.org/meetings/2018/09/votes

[wesbland]

Now that this is merged, it can be closed, right @schulzm / @tonyskjellum ?

[tonyskjellum]

@wesbland , @schulzm --- we could close this, BUT, given the work on semantic terms, we probably will be writing a clarifying follow-up ticket.

Persistent collective INIT calls will be clarified in that work. So... close the ticket or not?

[wesbland]

Entirely up to you. If you're going to keep working on this, you're welcome to continue on this issue so you don't lose the context. If you're going in a new direction, a new issue is fine too. There's no rules on this.

…

[tonyskjellum]

OK, thanks. Context concerns are why I want to hold off closing, yes.

…

On Wed, Mar 20, 2019 at 3:05 PM Wesley Bland ***@***.***> wrote: Entirely up to you. If you're going to keep working on this, you're welcome to continue on this issue so you don't lose the context. If you're going in a new direction, a new issue is fine too. There's no rules on this. > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA38iSNObhq5sIYloqs1SNiBbjygCkaKks5vYoaVgaJpZM4GxSUF> .

-- Anthony Skjellum, PhD [email protected] Cell: +1-205-807-4968

[wesbland]

@tonyskjellum I went ahead and closed this to help with my bookkeeping for MPI 4.0. Obviously, this issue won't disappear and any further work can have its own issue.