reuse client port number if possible

[amitmurthy]

This fixes a couple of issues that we ran into while launching a 1000 workers, with 64 workers to a node

• one, client side port number reuse, so as to not run out of the ephemeral port numbers used on the client side of a TCP connection.
• prevent runaway error messages logged when an accept call fails.

[vtjnash]

probably worth taking a look at libuv/libuv@a385ae4, where libuv explictly disabled this and the bug referenced there https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=174087

[amitmurthy]

Thanks.

Those links are about issues with SO_REUSEADDR. The option being set in this PR is SO_REUSEPORT which was introduced first in BSD and fairly recently in Linux (kernel 3.9 onwards).

This PR gets an ephemeral port number the first time around and then reuses the same port number for all subsequent connect request. This works because we only setup a single connection to each remote ip:port

The funny thing though is that on the client side, setting SO_REUSEPORT before bind throws an error, while setting it after results in the desired behaviour.

[jakebolewski]

you are throwing strings here

[jakebolewski]

👍

[yuyichao]

Is this warning during Travis CI related to this PR and is it expected?

[yuyichao]

It's in the CI run for this PR as well. @amitmurthy

[amitmurthy]

I forgot Travis runs on Ubuntu 12.04, so it is expected.

I'll modify the warning to show up only when we are adding a large number of workers, say > 128, since it is not really an issue below that.

[yuyichao]

Is that because the kernel is not new enough? and can it be detected?

[amitmurthy]

Yes. And yes.

[StefanKarpinski]

I'm seeing this during tests:

    JULIA test/all
WARNING: Unable to reuse port : SystemError("getsockname() : ",48)
WARNING: Unable to reuse port : SystemError("getsockname() : ",48)WARNING: Unable to reuse port : SystemError("getsockname() : ",48)WARNING: Unable to reuse port : SystemError("getsockname() : ",48)
WARNING: Unable to reuse port : SystemError("getsockname() : ",48)WARNING: Unable to reuse port : SystemError("getsockname() : ",48)
WARNING: Unable to reuse port : SystemError("getsockname() : ",48)

Julia Version 0.4.0-dev+6022
Commit f4eb92b (2015-07-17 13:41 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin14.3.0)
  CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[amitmurthy]

Working on this here - #12192

Looks like on OSX we need to setsockopt before bind (as sounds logical), but on Linux that did not work and bind followed by setsockopt worked.

Maybe I should limit setting this only for Linux since it is unlikely that folks on OSX will run with hundreds of workers?

[StefanKarpinski]

Nah, let's fix it right. Better to have things just work. Someone may want hundreds of workers on OS X.

[ScottPJones]

Maybe this is just because in my use cases, things are generally I/O bound, so I'm used to needing lots and lots of workers, even on a old 8-core Mac Pro, as opposed to the probably more CPU bound stuff that is more frequently done in numerical/scientific computing.
Please make it work right on OS X also.

[StefanKarpinski]

Let me know if you need help testing, @amitmurthy.

[amitmurthy]

Thanks @StefanKarpinski . I have access to a Mac and was trying it make it work there. The bad news is that I don't think I can make it work on OSX because we don't have a way to access the socket fd before a bind - see libuv/libuv#386 (comment) .

On Linux kernels > 3.9 setsockopt after bind just happens to work.

The ability to get access to a socket before bind landed fairly recently in mainstream libuv - libuv/libuv#400 .

Also SO_REUSEPORT is not supported on Windows.

I'll go ahead and make this a Linux only thing for now. Will revisit OSX when our libuv fork catches up with the above mentioned patch.

[StefanKarpinski]

Sounds good.