producer: connection backoff

[mreiferson]

In #95 we documented a case where nsq_to_nsq (which uses both Consumer and Producer) would panic due to a bug in the exit timeout code path.

The bug is exacerbated by the fact that in rare cases the producer could block serially for 1s per PUB if the remote address is black holing the connection attempts.

An alternative strategy might be to back off connection attempts and return errors instantly during backoff windows.

Thoughts?

[twmb]

This sounds good. For the case of nsq_to_nsq, the first error returned would also backoff the consumer side, which would be decent. The only thing that I don't like about returning error immediately for all messages is that message.Attempts would be incremented for all messages in flight, when, in reality, only the first message was truly attempted.

[mreiferson]

This gets back to my comment on nsqio/nsq#380 about the semantics of attempt - I would argue it is a failed attempt if you're in a backoff window!

[twmb]

I like to think of it as "oh hey, while consuming I got in a bad state, I'm going to set my RDY to 0 and also not even look at anything that was in flight".

Also, with current behavior, a nsq_to_nsq would only attempt, say, 8 before the backoff window is over, meaning only 8 have a higher attempt. If we fast track returned errors in backoff windows, then up to MaxInflight messages would get an attempt incremented, which is different behavior, which could mean potentially thousands of messages are one step closer to death.

[mreiferson]

For cases where you want max attempts, I don't think considering this edge case as an attempt to be a practical concern.

For other cases (where you want "infinite" attempts) it doesn't matter anyway.

[twmb]

I suppose that's true, which means this issue can go forward, but I still think that NoAttempt would be a useful addition :).