Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Embiggenment ("BigCount") #137

Closed
martinruefenacht opened this issue May 31, 2019 · 63 comments
Closed

The Embiggenment ("BigCount") #137

martinruefenacht opened this issue May 31, 2019 · 63 comments
Assignees
Labels
had reading Completed the formal proposal reading passed final vote Passed the final formal vote passed first vote Passed the first formal vote wg-large-counts Large Counts Working Group
Milestone

Comments

@martinruefenacht
Copy link

martinruefenacht commented May 31, 2019

Problem

Modern applications want to use more than (2^31-1) count arguments across the MPI Standard.

Large counts are possible, but users do not like constructing them with MPI datatypes. (#80)

Proposal

The high-level intent of this proposal is to allow users to call functions such as MPI_SEND with either "small" counts (e.g., int in C/C++ and INTEGER in Fortran) or "large" counts (e.g., MPI_Count in C/C++ and INTEGER,KIND=MPI_COUNT_KIND in Fortran) -- regardless as to whether the count parameter is IN, INOUT, or OUT -- and have MPI "do the right thing".
It will specifically be allowed that MPI_Count is actually int (e.g., for implementations or environments that do not support large counts as prescribed in this proposal).

Specifically: we do not want to expose alternate symbol names (e.g., MPI_SEND_X) to applications -- they should continue to use names such as MPI_SEND, and the language/MPI implementation will select the right back-end implementation depending on the type of the count parameter(s).

More detail

Providing both int and MPI_Count parameters for all relevant MPI functions in will use different mechanisms in C, C++, and Fortran.

  • C: C11 _Generic
    • Sidenote: supported in GCC since v4.9
  • C++: Function overloading
    • Sidenote: we cannot use the same _Generic mechanism because the C++ language does not support _Generic
  • Fortran: Generic functions
    • Sidenote: supported since F90
    • Sidenote: we will only be updating the mpi_f08 module to include MPI_Count functionality. Applications using mpif.h or the mpi modules will need to update to (perhaps piecemeal) use the mpi_f08 module to get access to MPI_Count-enabled functions.

Note that MPI-3.x supports C++ only indirectly: the C bindings are carefully crafted such that they work in both C and C++. This proposal will intentionally add a C++-specific mechanism (function overloading). Other than that, there will be a 1:1 global function set compared to the C bindings.

Note: we are electing to create the mpi.hpp header to the standard for use in C++ MPI applications. This is in recognition that C and C++ are continuing to diverge as languages, and we may need further separation between the C and C++ interfaces someday.

Also note that this proposal does not change the displacement parameters in v or w functions -- specifically because displacements and counts are separate concepts.

Changes to the Text

This will cause extensive changes across much of the existing standard. Conceptually:

  • We will change the language neutral bindings, C bindings, and Fortran bindings that contain "count" arguments to include an indicator that there are actually 2 such functions, one with "small" integer types and one with MPI_Count types.
  • Various other cleanups to recognise that functions such as MPI_SEND now have two versions.
  • Regulating the symbol names of these duplicated functions in order to support link-intercepting / PMPI-based tools.
  • The Language Bindings chapter will contain the main additions and the macros for the C and Fortran syntax will need to be changed.

Impact on Implementations

Implementations will have to provide the mpi.hpp header for C++ usage. Additional glue code will be required to support both MPI_Count and int interface level functions.

Tools will be required to support the new MPI symbols introduced by this proposal.

Impact on Users

The impact on users is kept to the minimum;

  • Correct, existing MPI applications will require no source code change.
    • One caveat: C++ applications will need to include mpi.hpp instead of mpi.h
  • ABI for existing applications should be able to be preserved by MPI implementations:
    • Existing C applications should be able to continue to use the same symbols as before (e.g., MPI_Send)
    • Existing C++ applications should be able to continue to use the same symbols as before (i.e., the C symbols -- e.g., MPI_Send). However, if C++ applications recompile with a _Generic-enabled mpi.h, they will need to update their source code to include mpi.hpp, and then they will be switched to use C++ symbols (i.e., a C++-compiler-munged version of MPI_Send)
    • Existing Fortran applications should be able to continue to use the same symbols as before (i.e., the ISO-C-mandated symbol names from the mpi_f08 module).
  • Applications can start using MPI_Count for count parameters, but they will need to update their source code.

References

Other tickets referring to this issue in some way or another: #80, #97, #98, #99, #100, #105, #108

Pull request: https://github.com/mpi-forum/mpi-standard/pull/268
(old pull request: https://github.com/mpi-forum/mpi-standard/pull/132 )

PDF file that passed first vote: mpi40-report-826f32a-ticket137-14jun20.pdf

The latest PDF file: mpi40-report-ticket137-cfed9a5-14Sep2020.pdf

PDF file with changes since first vote highlighted: mpi40-report-ticket137-cfed9a5-14Sep2020-highlighted.pdf

Please also see Froozle MPI, a no-op MPI implementation that shows one possible way that the proposals in this issue could be implemented.

@martinruefenacht
Copy link
Author

A pull request with specific text will be forthcoming.

@omor1
Copy link
Member

omor1 commented Jun 1, 2019

I see one immediate problem with this proposal. Currently, all MPI routines are explicitly defined as functions. This allows one to take the address of it to create a function pointer, for instance:

int (*MY_Send)(const void *, int, MPI_Datatype, int, int, MPI_Comm) = &MPI_Send;

A function-like macro does not support this use case, niche as it may be. There may be similar problems with the profiling interface as well, which is likely a greater concern.

@omor1
Copy link
Member

omor1 commented Jun 1, 2019

Additionally, there might be some strangeness around integer promotion rules and how they interact with _Generic. I haven't specifically tested this though.

@dholmes-epcc-ed-ac-uk
Copy link
Member

Corollary of @omor1's comment:

Can the function pointers proposal be composed as a library on top of current MPI definitions?

@jsquyres
Copy link
Member

jsquyres commented Jun 1, 2019

@omor1 See the slides I presented this past Thursday at the MPI Forum meeting: https://github.com/mpi-forum/mpi-forum.github.io/blob/master/slides/2019/05/2019-05-30-BigCount-solutions-for-MPI-4.pdf . One direct point from there is that yes, we have to standardize all the underlying symbol names for PMPI reasons.

But in short, yes, there are definite tradeoffs to every solution we have looked at. The user demand seems high to allow "big" count values, though. Hence, if the Forum wants to handle "BigCount" in C, tradeoffs will need to be made.

Some points not explicitly covered in the slide (but we talked about verbally):

  1. Yes, if you have a short variable and pass it to a C11 _Generic function that expects an int or MPI_Count (where those are two different types), you will get a compile-time error. That being said, there is also a default option for C11 _Generic that would allow you to pass such non-int/non-MPI_Count parameters. This allows "normal" promotion to take place.
  2. We will shortly be releasing a dummy, 6-function-like no-op MPI package that shows what all the symbols will look like so that people (e.g., tools authors) can look at this in detail. We can then use this package to iterate to approach a solution with the least unpalatable tradeoffs (i.e., use concrete code to show this stuff, not just text / words).

Are function pointers guaranteed to work? Interestingly enough, I just re-read the profiling section of MPI-3.1 and it does not guarantee that the back-end C symbols are actually the same as the names of the API functions (!). It just guarantees that name-shifted versions must be available (e.g., PMPI_Foo). The majority of the chapter has a giant disclaimer effectively saying "this is one way it could be done; there could be others." Meaning: it may well be valid to have #define MPI_Send Froozle_Send in mpi.h (!!). Indeed, it may even be valid to have MPI_Send, itself, to be a pointer, in which case foo = MPI_Send would work, but your example would fail (because you'd take the address of the pointer, not the value of the pointer). Meaning: my point in the slides of needing to standardize the back-end C symbol names may be above and beyond what MPI-3.1 requires! This is a surprise to me.

All that being said, FWIW, my testing with function pointers seem to work with C11 _Generic. This is a snipit from one of the examples in our upcoming no-op MPI package that uses C11 _Generic:

#include <mpi.h>

typedef int (*foo)(const void *buf, int count,
                   MPI_Datatype datatype,
                   int dest, int tag, MPI_Comm comm);

int bar(const void *buf, int count,
        MPI_Datatype datatype,
        int dest, int tag, MPI_Comm comm)
{
    printf("I am in bar!\n");
    return 0;
}

int main(int argc, char **argv)
{
    char buffer[SIZE];

    foo new_func = bar;

    MPI_Init(NULL, NULL);

    printf(">> The following functions should call MPI_Send (with int params)\n");
    int i = SIZE;
    MPI_Send(buffer, i, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    MPI_Send(buffer, 32, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    new_func(buffer, 32, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    // ...

This compiles and runs properly on my MacOS 10.14.5 laptop (i.e., when I call newfunc(), it prints I am in bar!).

@jsquyres
Copy link
Member

jsquyres commented Jun 1, 2019

Blah -- my original example was flawed. Here's what should be a correct one:

#include <mpi.h>

typedef int (*foo)(const void *buf, int count,
                   MPI_Datatype datatype,
                   int dest, int tag, MPI_Comm comm);

int main(int argc, char **argv)
{
    char buffer[SIZE];

    foo bar = MPI_Send;

    MPI_Init(NULL, NULL);
    printf("Calling function pointer\n");
    bar(buffer, 32, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    // ...

The bar() invocation calls the MPI_Send with int parameters (which is intended -- in our example no-op MPI, the back-end symbol with int params is still MPI_Send).

If we mandate the symbols for the MPI_Count-style count APIs, then you can also intercept those symbols, too (e.g., MPI_Send_count, or somesuch), and/or have function pointers for them (assuming you have the right type for the function pointers that uses MPI_Count, not int, for count params).

@omor1
Copy link
Member

omor1 commented Jun 1, 2019

Right, if you define the MPI_Send generic macro and name the 'regular' symbol MPI_Send as well it appears to work.

int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm);
int MPI_Send_x(const void *buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm);

#define MPI_Send(buf, count, datatype, dest, tag, comm) \
        _Generic((count),                               \
                default:  MPI_Send,                     \
                int:      MPI_Send,                     \
                size_t:   MPI_Send_x                    \
        )(buf, count, datatype, dest, tag, comm)

int main(int argc, char *argv[])
{
        int buffer[500];
        int count1 = 500;
        MPI_Count count2 = 500;

        MPI_Init(NULL, NULL);

        // call MPI_Send
        MPI_Send(buffer, count1, MPI_INT, 0, 0, MPI_COMM_WORLD);
        // call MPI_Send_x
        MPI_Send(buffer, count2, MPI_INT, 0, 0, MPI_COMM_WORLD);

        int (*My_Send)(const void *, int, MPI_Datatype, int, int, MPI_Comm) = MPI_Send;
        int (*MY_Send_x)(const void *, MPI_Count, MPI_Datatype, int, int, MPI_Comm) = MPI_Send_x;

        // ...
}

@hzhou
Copy link

hzhou commented Jun 2, 2019

Can this be solved by a "big count" binding -- e.g. mpi_x.h, in which, MPI_Send is defined as MPI_Send_x?

I am having a hard time imagine the need for an application to require access to both versions; therefore, exposing both versions to the user seems just more complicated than necessary. In fact, I think the implementation should just all move to MPI_Count internally and as long as it is bigger than int and not exceed native support, I don't see any problems. Then, instead of "big count" binding, we just need support a "legacy" binding.

@hzhou
Copy link

hzhou commented Jun 2, 2019

To think about it, the function pointer solution is very similar to the binding solution. One is runtime binding, the other is static binding.

I personally always preferred static solutions as they are simpler (to the user).

@tonyskjellum
Copy link

tonyskjellum commented Jun 2, 2019 via email

@jeffhammond
Copy link
Member

Please split this into three tickets:

  1. Fortran 90+ interfaces
  2. C++ overloads
  3. C11 _Generic

You will find that 3 is much harder than you think at first glance. C11 _Generic is not like C++ overloading.

Please also reconsider your position on displacements. If you do not fix displacements when you fix counts, you will end up with a half-broken, often useless interface. Please read the BigMPI paper if necessary.

@jeffhammond
Copy link
Member

Also, your desire to not support MPI_Send_x and friends is misguided, because you have to define those symbols for C11 _Generic to work. Furthermore, MPI has support for many languages -- Python, Rust, D, etc. -- and does so via the C89 bindings. If you attempt to hide explicit large-count symbols, you will make large-count support in these language interfaces impossible.

@omor1
Copy link
Member

omor1 commented Jun 3, 2019

Re @hzhou, if the interpretation that MPI never supported any of the non-PMPI functions as explicit symbols, then there should technically be little backwards compatibility issues unless someone explicitly declares int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); without including mpi.h (somehow).

Any int-sized parameters should undergo an implicit integer conversion to MPI_Count. The only backwards-compatibility problem I can see is if someone were to take the address of the symbol to obtain a function pointer—which as discussed above might not technically be valid according the standard in any case.

Since MPI never defined an ABI in the first place (e.g. under an ILP64 system then current MPI allows for 'big counts') and as far as I can tell API breakage would be minimal, I see no reason why it couldn't work.

@tonyskjellum
Copy link

tonyskjellum commented Jun 3, 2019 via email

@tonyskjellum
Copy link

tonyskjellum commented Jun 3, 2019 via email

jsquyres added a commit to jsquyres/froozle-mpi that referenced this issue Jun 3, 2019
A first cut of Froozle MPI, a no-op MPI implementation just to
experiment with MPI Forum issue #137
(mpi-forum/mpi-issues#137).

This commit includes a first cut of C and C++ APIs; Fortran is still
to come.

Signed-off-by: Jeff Squyres <[email protected]>
jsquyres added a commit to jsquyres/froozle-mpi that referenced this issue Jun 3, 2019
A first cut of Froozle MPI, a no-op MPI implementation just to
experiment with MPI Forum issue #137
(mpi-forum/mpi-issues#137).

This commit includes a first cut of C and C++ APIs; Fortran is still
to come.

Signed-off-by: Jeff Squyres <[email protected]>
@mhoemmen
Copy link

mhoemmen commented Jun 3, 2019

@tonyskjellum wrote:

story for later with our forthcoming proposal for a Language interface for python and C++ vs language binding on top of C89 bindings.

You'll probably get plenty of critique, but if you want an extra pair of eyes on the C++ binding proposal, I'll be happy to oblige.

@tonyskjellum
Copy link

tonyskjellum commented Jun 3, 2019 via email

@jeffhammond
Copy link
Member

jeffhammond commented Jun 3, 2019 via email

@omor1
Copy link
Member

omor1 commented Jun 4, 2019

Please cite the text of MPI 3.1 that mandates using the header and disallows using legacy C semantics that assume undeclared functions return int, which is correct for almost all MPI functions.

As far as I can tell, the MPI Standard never specifies a particular version of the C standard, only ever referring to "ISO C", which can variously refer to any version of ISO/IEC 9899, the most recent version (ISO/IEC 9899:2018), or the version originally ratified (ISO/IEC 9899:1990), depending on interpretation. In particular, I believe that assuming that function calls with an undeclared identifier are declared as extern int identifier(); is deprecated since C99. Though of course all major compilers will support it as an extension (and warn about it).

You'd still have the problem that the various MPI types (e.g. MPI_Comm at the most basic) are undeclared, so I don't know how you'd be able to use them.

@jsquyres
Copy link
Member

I have created Froozle MPI, a no-op MPI implementation that shows one possible way that the proposals in this issue could be implemented.

The intent for Froozle is to ground the discussions on this issue (and the eventual PR) in with some code, and provide a basis for testing, validation, etc.

I just released v0.5 (Froozle's first release).

Comments welcome.

@tonyskjellum
Copy link

tonyskjellum commented Jun 12, 2019 via email

@jeffhammond
Copy link
Member

$ gcc -std=c89 -c no-header.c && echo SUCCESS
SUCCESS
int main(int argc, char* argv[])
{
    int rc;
    rc = MPI_Init(&argc,&argv);
    rc = MPI_Finalize();
}

Obviously, I'd have to cheat if I want to compile with MPI_COMM_WORLD.

@hjelmn
Copy link

hjelmn commented Jun 12, 2019

@jeffhammond Honestly, that is just a straw-man example. Any attempt to actually use MPI functions without mpi.h would require the user to declare MPI symbols. I agree with @tonyskjellum in saying that, even though it isn't explicit in the standard, would make an erroneous program.

Example: A user could do something like declare:

typedef int MPI_Comm;
extern MPI_Comm MPI_COMM_WORLD;
extern int MPI_Barrier (MPI_Comm comm):

and it would only work with one of the two primary implementations. This is totally erroneous and if any user actually does that and we break their code so what? We shouldn't be wasting our time thinking about that case. If we need to update the text to say including mpi.h is required then lets do that.

@jeffhammond
Copy link
Member

We should change the standard to make mpi.h inclusion mandatory rather than making proclamations about what is implicitly prohibited by common sense.

We had a code that declared a user communicator as an int that worked without error for years because it was never compiled against Open-MPI.

@dholmes-epcc-ed-ac-uk
Copy link
Member

For those used to untyped languages, using auto is fairly natural - and it gets you one step further. It still feels like a straw-man, though.

int main() {
    auto myComm;
    auto myRank, mySize;
    MPI_Init();
    MPI_Comm_get_parent(&myComm);
    MPI_Comm_rank(myComm, &myRank);
    MPI_Comm_size(myComm, &mySize);
    if (0==myRank) {
    } else {
    }
    MPI_Finalize();
    return 0;
}
$ gcc -std=c89 -c no-header.c && echo SUCCESS
SUCCESS
int main(int argc, char* argv[])
{
    int rc;
    rc = MPI_Init(&argc,&argv);
    rc = MPI_Finalize();
}

Obviously, I'd have to cheat if I want to compile with MPI_COMM_WORLD.

@tonyskjellum
Copy link

tonyskjellum commented Jun 13, 2019 via email

@wesbland
Copy link
Member

The second vote for this at the August, 2020 meeting was postponed to the September, 2020 meeting.

@jeffhammond
Copy link
Member

I am going to start sending bills for anti-anxiety medication to the MPI Forum 😒

@dholmes-epcc-ed-ac-uk
Copy link
Member

@jeffhammond in which case, don't ... I repeat DON'T ... look at the Slack channel right now.

@jeffhammond
Copy link
Member

First, https://en.wikipedia.org/wiki/Streisand_effect :-)

Second, I looked and am not concerned. It looks like a bunch of smart people are working hard to do straightforward things that need to be done. I did not find any showstopper issues...

@dholmes-epcc-ed-ac-uk
Copy link
Member

I strongly recommend that this should be inspected by a real Fortran specialist on Fortran subroutine overloading through different abstract Interfaces which differ only in the signature of a procedure dummy argument.

Fortran 2008's 12.4.3.4.5 Restrictions on generic declarations:

Two dummy arguments are distinguishable if
1•one is a procedure and the other is a data object,
2•they are both data objects or known to be functions, and neither is TKR compatible with the other,
3•one has the ALLOCATABLE attribute and the other has the POINTER attribute, or
4•one is a function with nonzero rank and the other is not known to be a function.

Thus, the F08 bindings of MPI_Op_create_l and MPI_Register_datarep_l must be explicitly defined in the MPI standard since generic overloading is not possible (2 procedures cannot be distinguished).

I think, that these 2 exceptions should be mentioned in Section 2.6.2, Line 27-28.

Question: does this force us to introduce an Embiggened operator, MPI_OP_C, to distinguish between MPI_OP that represents a user-defined operator function with INTEGER :: len and MPI_OP_C that represents a user-defined operator function with INTEGER(KIND=MPI_COUNT_KIND) :: len?

This would mean that MPI_REDUCE would take MPI_OP whereas MPI_REDUCE_C would take MPI_OP_C. This is not needed by Fortran to distinguish MPI_REDUCE from MPI_REDUCE_C (because of the count parameter, which is TKR different) but does expressing all operators as MPI_OP cause implementation issues deep inside somewhere? I can imagine that MPI_OP might be an integer index into an internal array of operators, i.e. an array of function pointers. However, Fortran is strongly-typed so can such an array be declared correctly so that it can contain function pointers with different signatures?

To be clear, I really don’t want to introduce MPI_OP_C (because built-in operators!) but I think we need a proof-of-concept implementation to reassure us that this is not made necessary by Fortran’s strong-typing rules.

** To be discussed at the virtual meeting today **

@hritzdorf
Copy link

hritzdorf commented Aug 26, 2020 via email

@wesbland
Copy link
Member

This failed a no-no vote on 2020-09-28.

https://www.mpi-forum.org/meetings/2020/09/votes

@naughtont3
Copy link

Sept-2020 Meeting detailed list:

@wesbland
Copy link
Member

wesbland commented Oct 1, 2020

This passed a second vote on 2020-09-30.

https://www.mpi-forum.org/meetings/2020/09/votes

The forum agreed to resolve all remaining issues as an errata vote at the December 2020 meeting.

@dholmes-epcc-ed-ac-uk
Copy link
Member

@wgropp To be clear: no PR should be merged in its current state in response to the successful final vote for this issue.

This 2nd vote accepted commit 826f32a (https://github.com/mpi-forum/mpi-standard/pull/132/commits/826f32a47a1a15e4ace31609ab268e04bbca45b7), which is in the history of PR 132 (https://github.com/mpi-forum/mpi-standard/pull/132) but is not the HEAD of the branch behind that PR. I also created PR 294 (https://github.com/mpi-forum/mpi-standard/pull/294) to point directly to this voter-for commit, for easier reference.

The proposed errata for the December meeting is represented by PR 268 (https://github.com/mpi-forum/mpi-standard/pull/268), which is rebased from a different point in the mpi-4.x history and is difficult to compare directly to PR 132. All changes in PR 132 (including those beyond the voted-for commit) are included in the errata PR 268 along with additional fixes that were the subject of the no-no vote that failed in the Sept 2020 meeting and will be the subject of the errata proposal in the Dec 2020 meeting.

The plan is to merge these voted-for changes and the to-be-voted-for errata atomically by merging PR 268 once the errata vote has also passed. The intent is to iterate on the errata until it passes (or until the Forum decides to reverse the final vote for this issue). Recent comments and merge conflicts are still TBD and will be addressed on PR 268.

@wgropp
Copy link

wgropp commented Oct 2, 2020

@dholmes-epcc-ed-ac-uk thanks for the clear statement - I will leave this one, and will work with @wesbland as discussed at the Forum meeting.

@dholmes-epcc-ed-ac-uk
Copy link
Member

The Embiggening(tm) is merged into mpi-4-rc!

https://files.slack.com/files-pri/TM2LH9685-F01BWJYMRE0/done.gif
HT @jeffhammond

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
had reading Completed the formal proposal reading passed final vote Passed the final formal vote passed first vote Passed the first formal vote wg-large-counts Large Counts Working Group
Projects
None yet
Development

No branches or pull requests