Restrict interface bindings and test connections

[amitmurthy]

This PR does the following:

• addprocs restricts socket bindings to 127.0.0.1, the default ip or a specified bind-to address. Currently we listen on all interfaces.
• addprocs(N) by default binds only to 127.0.0.1.
• addprocs(N); addprocs(["some_host"]) will no longer work.
• addprocs(N; restrict=false); addprocs(["some_host"]) is the alternative for adding local workers and then adding workers from a remote host.
• SSHManager now binds only to the specified ip or to the first interface ip returned by getipaddr.
• A cluster now has a "cookie" which is shared by all processes. The master generates a random string, which is then passed to the workers via startup parameter --worker <cookie>.
• All incoming socket connections are validated by ensuring that the first message received has the cluster cookie.

This PR addresses basic hygiene by avoiding unnecessary socket bindings and testing for connections. Full fledged security requirements are best addressed through a custom ClusterManager.

Doc updates are pending.

[tkelman]

If this gets backported, does this mean every worker will have to be using the very latest version of julia to be able to communicate with the main node? What about workers that get updated but node 1 still uses an older point version of julia? Would both of those situations break?

[amitmurthy]

Yes.

ClusterManagers.jl will also have to be updated to support this. Any other custom ClusterManagers would break too.

We could port only the first commit, i.e., binding only to one interface to 0.4. That would only break the specific use case of a local addprocs followed by a remote one. Which can be fixed by specifying restrict=false.

[amitmurthy]

Ready for review.

[tkelman]

should this really be exported? it's kind of an internal detail

[amitmurthy]

Only required by folks writing their own ClusterManagers. But I guess we could document its use as Base. cluster_cookie()

[tkelman]

better to do this inline rather than adding to this file

[amitmurthy]

Had it inline but moved it after removing the export since we cannot have the function definition as Base.cluster_cookie.

[amitmurthy]

Merging this in a short while if there are no major concerns.

[tkelman]

I wish you hadn't squashed this since the cookie part is not backwards compatible. Please open a separate PR against release-0.4 with the less breaking portion.

[amitmurthy]

Will keep this in mind in the future. For this patch though the first commit wouldn't have merged cleanly and we also need to update the manual.