More tolerance for notifications keep alive timeout

[tomaka]

I see this situation arising in the logs:

• The behaviour sends Open to the handler.
• The handler answers with OpenResultErr.
• The behaviour receives the OpenResultErr and immediately tries again by sending Open again.
• However, since the handler has transitioned to the Closed state after returning OpenResultErr, its keep-alive timeout kicks in and shuts down the connection.

While it is questionable to immediately try opening the same connection immediately again it has failed to open, it is clearly not intended that the keep-alive timeout triggers in the millisecond or so during which the handler is very temporarily in the Closed state.

This PR modifies the tolerance level in order to wait for 5 seconds after entering the Closed state, instead of just the first time after opening a connection.

[romanb]

So this is mainly an optimisation to avoid closing and re-opening connections or does the right "timing" in this sense actually cause a problem? The latter would be unfortunate, as no choice for the timeout could then guarantee that the problem does not arise, if I understand correctly.

[tomaka]

Yes, this is intended as an optimization.

There is indeed a problem happening, which is why an OpenResultErr is received. But this problem is caused by #7638 on the other side of the connection, and not by the connection being closed.

[tomaka]

bot merge

[ghost]

Waiting for commit status.

[ghost]

Checks failed; merge aborted.

[tomaka]

reconnect_after_disconnect stalled on CI. The test passes locally for me. Restarted CI to see whether it's a heisenbug.

[tomaka]

Of course the test passes on CI if I add logging.

[tomaka]

I just ran the test 100 times successfully in a row on my machine.

[gnunicorn]

is this still being worked on?

[tomaka]

No, I'm completely clueless as to why CI fails.
All tests pass smoothly locally. CI becomes green whenever I add any sort of logging.

[gnunicorn]

closing for a lack of progress.