Supporting multiple cluster schedulers

[ViralBShah]

Currently, supporting multiple cluster schedulers requires hacking code into multi.jl. It would be nice to have a way to abstract this out of Base. I would ideally like to have the base just have capabilities to ssh and launch jobs.

SGE and scyld support (#3546) should ideally be in packages, and extend the capabilities in base. This will require some re-thinking of the way the code to launch workers is organized currently. Once this is done, it will become much easier to support many other batch schedulers.

cc: @nlhelper

[amitmurthy]

👍

[ViralBShah]

Support for Microsoft cluster:

https://groups.google.com/forum/?fromgroups=#!topic/julia-users/H7-NReF7WBo

[ViralBShah]

It would be also nice to be able to allocate EC2 computers in the same way.

[pao]

@ViralBShah you're looking for @nlhepler, I think.

[amitmurthy]

A suggested design:

• Currently, the Worker processes write out the host:port they are listening on to stdout.
  This is parsed by the master process and connections established.
  Also the STDOUT of the worker processes is redirected to the Julia console.
• We can have another addprocs in Base:
• addprocs(n::Integer, launch_cb::Function, args...)
• The callback function is of the type launch_cb(julia_exe_name, julia_exe_args, args...) => io || (io, host, port) || (io, host) || (host, port)
• The launch_cb is called by addprocs to launch the Julia worker process on any machine in the cluster.
• The first 2 parameters - julia_exe_name, julia_exe_args - are specified by Base.addprocs. Remaining parameters
  are a pass through
• launch_cb can return dfferent combinations of io, host and port
• io is the stdout of the worker process. host and port is the address to use to connect to the process.
• If only an io is returned, it is parsed for the host port.
• If only host, port are returned they are used to connect to.
• If all three are returned, the io is read and then redirected to the console. host, port returned are used.
• if io, host is returned, the port is obtained from the io while the host returned is used.
• We can bundle existing sge, scyld and EC2 addprocs into a new package called Cluster,
  with appropriate callback functions for each cluster model.
• Usage in Julia would be

addprocs(5, Cluster.ec2, <args specific to EC2>)
addprocs(5, Cluster.sge, <args specific to sge>)
addprocs(5, Cluster.scyld, <args specific to scyld>)

Other clustering technologies can be supported via additions to the Cluster package.

[JeffBezanson]

That looks good.

[papamarkou]

👍

[daviddelaat]

For the API it would maybe be nice to make EC2, SGE, etc immutable types inheriting a common abstract cluster type, and then do:

addprocs(5, Cluster.EC2(<args specific to EC2>))

[amitmurthy]

Good idea.

I am thinking of extending the the signature of

addprocs(machines::AbstractVector;   tunnel=false, dir=JULIA_HOME, exename="./julia-release-basic", sshflags::Cmd=``)

to

addprocs(machines::Union(AbstractVector, ClusterManager);  tunnel=false, dir=JULIA_HOME, exename="./julia-release-basic", sshflags::Cmd=``)

where the concrete implementation of ClusterManager must have two members, viz., np - the number of procs to be added and launch - a function that will launch a worker instance.