Add write_timeout policy configuration option

[neilalexander]

This adds a new write_timeout configuration option which controls the slow consumer behaviour more precisely with what happens when the write_deadline is reached. This option can be set for clients, routes, gateways or leafnodes independently. Three values are supported:

• default: functionality as it stands today, retry for route/gateway/leafnode, close for client
• retry: when hitting the write_deadline, keep retrying writes until either we succeed or the ping interval hits
• close: when hitting the write_deadline, close the connection immediately

Signed-off-by: Neil Twigg neil@nats.io

[derekcollison]

LGTM

[derekcollison]

Can we store this once in the server struct?

[neilalexander]

Earlier versions of this code did have them in the server struct but the switch was slower than having it per-client, although I have noticed that I can make the WriteTimeoutPolicy a uint8 instead and modify the placement in the outbound struct, and then it actually takes up no more extra space due to existing alignment, so I'll do that.

[neilalexander]

main branch:

  // client.go:330 | Size: 112
  type outbound struct {
    nb  net.Buffers    ■ ■ ■ ■ ■ ■ ■ ■
                       ■ ■ ■ ■ ■ ■ ■ ■
                       ■ ■ ■ ■ ■ ■ ■ ■
    wnb net.Buffers    ■ ■ ■ ■ ■ ■ ■ ■
                       ■ ■ ■ ■ ■ ■ ■ ■
                       ■ ■ ■ ■ ■ ■ ■ ■
    pb  int64          ■ ■ ■ ■ ■ ■ ■ ■
    fsp int32          ■ ■ ■ ■ □ □ □ □
    sg  *sync.Cond     ■ ■ ■ ■ ■ ■ ■ ■
    wdl time.Duration  ■ ■ ■ ■ ■ ■ ■ ■
    mp  int64          ■ ■ ■ ■ ■ ■ ■ ■
    lft time.Duration  ■ ■ ■ ■ ■ ■ ■ ■
    stc chan struct{}  ■ ■ ■ ■ ■ ■ ■ ■
    cw  *s2.Writer     ■ ■ ■ ■ ■ ■ ■ ■
  }

After this fix, so same amount of space consumed overall:

  // client.go:350 | Size: 112
  type outbound struct {
    nb  net.Buffers         ■ ■ ■ ■ ■ ■ ■ ■
                            ■ ■ ■ ■ ■ ■ ■ ■
                            ■ ■ ■ ■ ■ ■ ■ ■
    wnb net.Buffers         ■ ■ ■ ■ ■ ■ ■ ■
                            ■ ■ ■ ■ ■ ■ ■ ■
                            ■ ■ ■ ■ ■ ■ ■ ■
    pb  int64               ■ ■ ■ ■ ■ ■ ■ ■
    fsp int32               ■ ■ ■ ■
    wtp WriteTimeoutPolicy          ■ □ □ □
    sg  *sync.Cond          ■ ■ ■ ■ ■ ■ ■ ■
    wdl time.Duration       ■ ■ ■ ■ ■ ■ ■ ■
    mp  int64               ■ ■ ■ ■ ■ ■ ■ ■
    lft time.Duration       ■ ■ ■ ■ ■ ■ ■ ■
    stc chan struct{}       ■ ■ ■ ■ ■ ■ ■ ■
    cw  *s2.Writer          ■ ■ ■ ■ ■ ■ ■ ■
  }

[derekcollison]

LGTM - thanks.