ETCD-180: Bug 1927942: UPSTREAM: <carry>: *: add support for socket options #70

hexfusion · 2021-03-09T21:32:35Z

This is a manual backport of etcd-io#12702

This PR adds support for setting socket options SO_REUSEADDR and SO_REUSEPORT to etcd listeners via ListenConfig. These options give the flexibility to cluster admins who wish to more explicit control of these features. What we have found is during etcd process restart there can be a considerable time waiting for the port to release as it is held open by TIME_WAIT which on many systems is 60s.

--socket-reuse-port: enables SO_REUSEPORT [1]  allows rebind of a port already in use. 
--socket-reuse-address: enables SO_REUSEADDR [1]  allows binding to an address in `TIME_WAIT` state.

$ ps aux | grep etcd | wc -l
0

$  ss --numeric -o state time-wait 
tcp    0       0                 10.0.138.12:34642            10.0.138.12:2380   timer:(timewait,50sec,0)                                                       
tcp    0       0                 10.0.138.12:34470            10.0.138.12:2380   timer:(timewait,35sec,0)                                                       
tcp    0       0                 10.0.138.12:34086            10.0.138.12:2380   timer:(timewait,10sec,0)                                                       
tcp    0       0                 10.0.138.12:34232            10.0.138.12:2380   timer:(timewait,20sec,0)                                                       
tcp    0       0                 10.0.138.12:34694            10.0.138.12:2380   timer:(timewait,55sec,0)                                                       
tcp    0       0                 10.0.138.12:34332            10.0.138.12:2380   timer:(timewait,25sec,0)                                                       
tcp    0       0                 10.0.138.12:33966            10.0.138.12:2380   timer:(timewait,845ms,0)                                                       
tcp    0       0                 10.0.138.12:34040            10.0.138.12:2380   timer:(timewait,5.845ms,0)                                                     
tcp    0       0                 10.0.138.12:34184            10.0.138.12:2380   timer:(timewait,15sec,0)                                                       
tcp    0       0                 10.0.138.12:34570            10.0.138.12:2380   timer:(timewait,45sec,0)                                                       
tcp    0       0                 10.0.138.12:34512            10.0.138.12:2380   timer:(timewait,40sec,0)                                                       
tcp    0       0                 10.0.138.12:34398            10.0.138.12:2380   timer:(timewait,30sec,0)

So we can wait for many seconds even after etcd process is long dead for client and peer ports to become available.

2021-02-19 13:57:25.117257 C | etcdmain: listen tcp 10.0.138.12:2380: bind: address already in use

A similar approach has been taken with the addition of these options in the kube-apiserver[2]

[1] https://man7.org/linux/man-pages/man7/socket.7.html
[2] kubernetes/kubernetes#93861

openshift-ci-robot · 2021-03-09T21:32:36Z

@hexfusion: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

UPSTREAM: : *: add support for socket options

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2021-03-09T21:56:47Z

@hexfusion: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

UPSTREAM: : *: add support for socket options

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2021-03-09T22:04:51Z

@hexfusion: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

UPSTREAM: : *: add support for socket options

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2021-03-10T01:30:43Z

etcd appears fine

/retest

hexfusion · 2021-03-10T01:32:09Z

/test openshift-perfscale-e2e-configmap-scale

openshift-ci-robot · 2021-03-10T01:32:20Z

@hexfusion: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test configmap-scale
/test e2e-aws
/test e2e-aws-serial
/test e2e-aws-upgrade
/test images
/test unit

Use /test all to run the following jobs:

pull-ci-openshift-etcd-openshift-4.8-e2e-aws
pull-ci-openshift-etcd-openshift-4.8-e2e-aws-serial
pull-ci-openshift-etcd-openshift-4.8-e2e-aws-upgrade
pull-ci-openshift-etcd-openshift-4.8-images
pull-ci-openshift-etcd-openshift-4.8-unit

In response to this:

/test openshift-perfscale-e2e-configmap-scale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2021-03-10T01:32:39Z

/test configmap-scale

hexfusion · 2021-03-10T20:23:19Z

/retest

openshift-ci-robot · 2021-03-10T21:26:35Z

@hexfusion: This pull request references Bugzilla bug 1927942, which is invalid:

expected Bugzilla bug 1927942 to depend on a bug targeting a release in 4.9.0 and in one of the following states: MODIFIED, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1927942: UPSTREAM: : *: add support for socket options

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2021-03-10T21:28:06Z

the bug is valid we have issues because we are a fork, manually setting group lead approved as it is not required in 4.8.

hexfusion · 2021-03-10T21:48:30Z

graceful term failure is unrelated to this change, unless the flag is enabled we should not see any direct impact on listeners.

https://search.ci.openshift.org/?search=kube-apiserver+terminates+within+graceful+termination+period+&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

/test e2e-aws

hexfusion · 2021-03-10T21:56:21Z

/refresh

openshift-ci-robot · 2021-03-10T21:56:30Z

@hexfusion: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws	`22e5a7c`	link	`/test e2e-aws`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hexfusion · 2021-03-10T21:57:21Z

/retest

pkg/transport/timeout_dialer.go

Signed-off-by: Sam Batschelet <[email protected]>

hexfusion · 2021-03-14T16:12:20Z

/retest

hexfusion · 2021-03-15T15:18:16Z

/retest

hexfusion · 2021-03-17T15:46:16Z

/retest

hexfusion · 2021-03-20T11:32:52Z

/test unit

hexfusion · 2021-03-22T13:37:39Z

/retest

hexfusion · 2021-03-23T03:10:32Z

/retest

hexfusion · 2021-03-27T17:35:26Z

/retest

hexfusion · 2021-03-29T12:32:26Z

/hold cancel

hexfusion · 2021-03-29T12:32:34Z

/retest

hexfusion · 2021-03-29T12:32:49Z

/hold

hexfusion · 2021-03-29T14:01:24Z

/hold cancel

openshift-bot · 2021-03-29T15:12:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2021-03-29T16:00:24Z

/retest

hexfusion · 2021-03-29T16:31:23Z

flake

/retest

openshift-bot · 2021-03-29T16:58:33Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2021-03-29T21:40:22Z

/test e2e-aws

openshift-bot · 2021-03-29T22:36:15Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2021-03-29T23:47:09Z

/retest

openshift-bot · 2021-03-29T23:54:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2021-03-30T01:41:46Z

/test e2e-aws-upgrade

hexfusion · 2021-03-30T04:01:25Z

/retest

openshift-bot · 2021-03-30T12:15:15Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2021-03-30T12:20:05Z

/retest

openshift-ci-robot · 2021-03-30T14:03:10Z

@hexfusion: Some pull requests linked via external trackers have merged:

etcd-io/etcd#12702

The following pull requests linked via external trackers have not merged:

openshift/cluster-etcd-operator#553 is open

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state. Once unlinked, request a bug refresh with /bugzilla refresh.

Bugzilla bug 1927942 has not been moved to the MODIFIED state.

In response to this:

ETCD-180: Bug 1927942: UPSTREAM: : *: add support for socket options

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from smarterclayton and wking March 9, 2021 21:33

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 9, 2021

hexfusion force-pushed the carry-etcd-12702 branch from 86d5c9d to 22e5a7c Compare March 9, 2021 21:51

hexfusion mentioned this pull request Mar 10, 2021

Bug 1927942: pkg/etcdenvvar: enable SO_REUSEADDR openshift/cluster-etcd-operator#553

Merged

hexfusion changed the title ~~UPSTREAM: <carry>: *: add support for socket options~~ Bug 1927942: UPSTREAM: <carry>: *: add support for socket options Mar 10, 2021

openshift-ci-robot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Mar 10, 2021

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Mar 10, 2021

retroflexer reviewed Mar 10, 2021

View reviewed changes

pkg/transport/timeout_dialer.go Outdated Show resolved Hide resolved

hexfusion force-pushed the carry-etcd-12702 branch 2 times, most recently from e98f920 to f0be2ed Compare March 10, 2021 22:24

hexfusion added 2 commits March 10, 2021 17:31

UPSTREAM: <carry>: *: add support for socket options

37c27cc

Signed-off-by: Sam Batschelet <[email protected]>

vendor: tidy

0e2dafd

Signed-off-by: Sam Batschelet <[email protected]>

hexfusion changed the title ~~Bug 1927942: UPSTREAM: <carry>: *: add support for socket options~~ ETCD-180: Bug 1927942: UPSTREAM: <carry>: *: add support for socket options Mar 22, 2021

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 29, 2021

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 29, 2021

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 29, 2021

openshift-merge-robot merged commit d031eea into openshift:openshift-4.8 Mar 30, 2021

hexfusion deleted the carry-etcd-12702 branch April 1, 2021 20:00

ETCD-180: Bug 1927942: UPSTREAM: <carry>: *: add support for socket options #70

ETCD-180: Bug 1927942: UPSTREAM: <carry>: *: add support for socket options #70

Conversation

hexfusion commented Mar 9, 2021 • edited Loading

openshift-ci-robot commented Mar 9, 2021

openshift-ci-robot commented Mar 9, 2021

openshift-ci-robot commented Mar 9, 2021

hexfusion commented Mar 10, 2021

hexfusion commented Mar 10, 2021

openshift-ci-robot commented Mar 10, 2021

hexfusion commented Mar 10, 2021

hexfusion commented Mar 10, 2021

openshift-ci-robot commented Mar 10, 2021

hexfusion commented Mar 10, 2021

hexfusion commented Mar 10, 2021

hexfusion commented Mar 10, 2021

openshift-ci-robot commented Mar 10, 2021

hexfusion commented Mar 10, 2021

hexfusion commented Mar 14, 2021

hexfusion commented Mar 15, 2021

hexfusion commented Mar 17, 2021

hexfusion commented Mar 20, 2021

hexfusion commented Mar 22, 2021

hexfusion commented Mar 23, 2021

hexfusion commented Mar 27, 2021

hexfusion commented Mar 29, 2021

hexfusion commented Mar 29, 2021

hexfusion commented Mar 29, 2021

hexfusion commented Mar 29, 2021

openshift-bot commented Mar 29, 2021

hexfusion commented Mar 29, 2021

hexfusion commented Mar 29, 2021

openshift-bot commented Mar 29, 2021

hexfusion commented Mar 29, 2021

openshift-bot commented Mar 29, 2021

hexfusion commented Mar 29, 2021

openshift-bot commented Mar 29, 2021

hexfusion commented Mar 30, 2021

hexfusion commented Mar 30, 2021

openshift-bot commented Mar 30, 2021

hexfusion commented Mar 30, 2021

openshift-ci-robot commented Mar 30, 2021 • edited Loading

hexfusion commented Mar 9, 2021 •

edited

Loading

openshift-ci-robot commented Mar 30, 2021 •

edited

Loading