Notes 2016 09 08

Participants

• Jean-Baptiste Besnard
• Ralph Castain
• John DelSignore
• Daniel Everett
• Marc-Andre Hermanns
• Bob Moench
• Kathryn Mohror
• Soren Rasmussen
• Martin Schulz
• Anh Vo

Notes

MPIR vs PMIx

• We need to figure out whether we will create a new MPIR-2 (process discovery interface) or if we will use the functionality already provided in the PMIx interface
  • We need to decide this because we've been debating it a long time and it would be good to move on if we are not going to develop it
• Is it simply a 'beauty contest' or are there technical aspects that make one option better than the other?
• A question raised is: What if PMIx is not as pervasive as we might hope?
  • Fragmentation might occur in debugger implementations where they need to support multiple interfaces to find processes
  • This would be an argument for an abstraction at the MPI layer
  • If there is an abstraction, then the implementation can fall back to whatever is available (e.g. MPIR1)
  • This is nice from the debugger perspective because they only have to support one interface
  • MPIR 1 was problematic because it was so implementation specific, a mess
  • Whatever we do, we don't want to dictate implementation
• Cray has a couple interfaces ALPS and CTI (Cray tools interface)
  • What if Cray doesn't use PMIx but uses CTI?
• Main concern is portability and implementation neutrality
• The weirdness comes in that the MPIR functionality is really partially resource manager functionality and partially MPI functionality
  • 2 pieces:
    • process discovery and tool launching: which processes are in the parallel job and where are they? Co-locate daemons there, etc.
      • parallel run doesn't have to be an MPI job, e.g. can run anything with SLURM/srun
    • assigning meaning to process entities: What MPI rank does this PID have? etc.
      • resource manager doesn't know about MPI process mapping, just the PIDs and nodes and tool daemons
      • MPI can assign this meaning
• mpirun is a very common way (perhaps the most common) of starting parallel jobs
  • mpirun launches daemons and the daemons launch processes
    • e.g. Open MPI, launches orte daemons that in turn start the MPI processes
  • In this case, the resource manager knows nothing of the MPI processes, only the daemons
  • problem is that the starter process (mpirun) is the sole owner of the proctable, the only source is mpirun
• How do we make this MPI and resource manager neutral?
• Isn't that the whole point of PMIx? Resource manager, program neutrality, just provide an interface
  • PMIx group has been pushing for community adoption and is getting good headway
  • Do we really want to do the same thing? Only later?
• The reality is that mpirun is not going away any time soon, so in a sense the 2 pieces of functionality for starting MPI processes are kind of tangled up (process discovery and assigning meaning)
  • especially since the starting of MPI processes can be done such that the resource manager doesn't know about them (the Open MPI daemon way)
  • So no matter what, we have to talk to the starter process
  • The PMIx group has been trying to make the interaction with the starter process the same across RMs
  • So that MPI implementations don't have to switch case everything depending on which system they are on
• What about threaded MPIs? Is this a technical issue that helps us decide this?
  • Debugger needs to find all MPI processes. right now this is done through proctable
  • then it's a separate mapping of objects in the program to some higher level meaning
  • this is a problem in the dynamic cases, sessions, endpoints, where entity membership can change
• We went through the PMIx interface again as a refresher
  • We decided that it provides the needed functionality, assuming it exists everywhere
  • Only problem is the prefix (hahaha)
• No consensus reached
• We will bring this up at the Forum in a plenary to hopefully get more opinions from implementors