Handshake backoff, when other node is not there #389
Description
E.g. spawn a node and have it join a not existing node, currently we are intensely retrying to connect:
2020-01-22T18:41:56+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
2020-01-22T18:41:56+0900 info: [sact://System@localhost:8228][ClusterShell.swift:620][thread:396395][/system/cluster] Extending handshake offer to sact://System@localhost:7337)
2020-01-22T18:41:56+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
2020-01-22T18:41:56+0900 info: [sact://System@localhost:8228][ClusterShell.swift:620][thread:396395][/system/cluster] Extending handshake offer to sact://System@localhost:7337)
2020-01-22T18:41:56+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
2020-01-22T18:41:57+0900 info: [sact://System@localhost:8228][ClusterShell.swift:620][thread:396395][/system/cluster] Extending handshake offer to sact://System@localhost:7337)
2020-01-22T18:41:57+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
2020-01-22T18:41:57+0900 info: [sact://System@localhost:8228][ClusterShell.swift:620][thread:396395][/system/cluster] Extending handshake offer to sact://System@localhost:7337)
2020-01-22T18:41:57+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
2020-01-22T18:41:57+0900 info: [sact://System@localhost:8228][ClusterShell.swift:620][thread:396395][/system/cluster] Extending handshake offer to sact://System@localhost:7337)
2020-01-22T18:41:57+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
2020-01-22T18:41:57+0900 info: [sact://System@localhost:8228][ClusterShell.swift:620][thread:396395][/system/cluster] Extending handshake offer to sact://System@localhost:7337)
2020-01-22T18:41:57+0900 warning: [sact://System@localhost:8228][ClusterShell.swift:753][thread:396395][/system/cluster] Failed await for outbound channel to sact://System@localhost:7337; Error was: NIOConnectionError(host: "localhost", port: 7337, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:7337, error: connection reset (error set): Connection refused (errno: 61))])
This should be on a backoff and max retries amount