Bug 1984683: use new default leader election values to handle apiserver rollout on SNO by creydr · Pull Request #328 · openshift/sdn

creydr · 2021-07-30T06:46:42Z

In case of SDN gets deployed on SNO, it can run into leader election timeouts during an api server rollout (as only one api server is available and it doesn't restart fast enough yet).
Therefor the default values for leader election have been updated (openshift/library-go#1104) to the recommended values.

This PR updates the version of the openshift/library-go library to use these new values and being able to handle a 60 seconds downtime of the api server on SNO.

How to reproduce / validate:

Start a SNO cluster with this patch
Trigger a apiserver rollout (e.g. via k patch kubeapiserver/cluster --type merge -p '{"spec":{"forceRedeploymentReason":"something"}}')
Check the logs from sdncontroller pod during the update
Validate the pod did not get restarted cause it lost the lease (e.g. check no previous logs are available / doesn't contain leaderelection lost

Special notes for reviewers:

had to update the go version for the build root image:

sdn/.ci-operator.yaml

Lines 1 to 4 in d979606

    
           build_root_image: 
        
             name: release 
        
             namespace: openshift 
        
             tag: golang-1.16

as openshift/build-machinery required at least go 1.15.2.

openshift-ci · 2021-07-30T06:46:48Z

@creydr: This pull request references Bugzilla bug 1984683, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

Details

In response to this:

Bug 1984683: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-07-30T06:46:50Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

creydr · 2021-07-30T11:36:04Z

/retest

Signed-off-by: Christoph Stäbler <cstabler@redhat.com>

creydr · 2021-07-30T12:22:58Z

/retest

openshift-ci · 2021-07-30T12:23:51Z

@creydr: This pull request references Bugzilla bug 1984683, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

Details

In response to this:

Bug 1984683: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-07-30T13:49:58Z

/retest-required

creydr · 2021-07-30T15:52:34Z

/retest-required

creydr · 2021-07-30T23:25:20Z

/retest-required

creydr · 2021-08-02T06:45:37Z

@zhaozhanqi, @abhat, @alexanderConstantinescu could you have a look at this PR please?
Thanks

zhaozhanqi · 2021-08-02T10:37:01Z

@zhaozhanqi, @abhat, @alexanderConstantinescu could you have a look at this PR please?
Thanks

I added comment in https://bugzilla.redhat.com/show_bug.cgi?id=1984683#c8 please help check, thanks

creydr · 2021-08-03T14:59:30Z

@abhat could you have a look at this PR?

abhat · 2021-08-03T15:45:19Z

/lgtm

abhat · 2021-08-03T15:45:28Z

/approve

openshift-ci · 2021-08-03T15:46:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, creydr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2021-08-03T15:46:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, creydr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-08-04T04:26:23Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-04T04:39:21Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-04T06:04:17Z

@creydr: All pull requests linked via external trackers have merged:

openshift/sdn#328

Bugzilla bug 1984683 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1984683: use new default leader election values to handle apiserver rollout on SNO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested a review from zhaozhanqi July 30, 2021 06:46

openshift-ci bot requested review from abhat and alexanderConstantinescu July 30, 2021 06:47

creydr marked this pull request as ready for review July 30, 2021 07:48

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2021

creydr marked this pull request as draft July 30, 2021 07:57

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2021

creydr force-pushed the use-new-leader-election-values-to-handle-60-sec-apiserver-rollout-in-sno branch from e3b0dbe to 164424c Compare July 30, 2021 09:55

creydr force-pushed the use-new-leader-election-values-to-handle-60-sec-apiserver-rollout-in-sno branch 2 times, most recently from 995c706 to 6402e7f Compare July 30, 2021 12:08

Bump openshift/library-go

d979606

Signed-off-by: Christoph Stäbler <cstabler@redhat.com>

creydr force-pushed the use-new-leader-election-values-to-handle-60-sec-apiserver-rollout-in-sno branch from 6402e7f to d979606 Compare July 30, 2021 12:22

creydr marked this pull request as ready for review July 30, 2021 12:24

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 30, 2021

openshift-ci bot assigned abhat Aug 3, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 3, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 3, 2021

openshift-ci bot merged commit a7a08ff into openshift:master Aug 4, 2021

	build_root_image:
	name: release
	namespace: openshift
	tag: golang-1.16

Conversation

creydr commented Jul 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Jul 30, 2021

Uh oh!

openshift-ci bot commented Jul 30, 2021

Uh oh!

creydr commented Jul 30, 2021

Uh oh!

creydr commented Jul 30, 2021

Uh oh!

openshift-ci bot commented Jul 30, 2021

Uh oh!

creydr commented Jul 30, 2021

Uh oh!

creydr commented Jul 30, 2021

Uh oh!

creydr commented Jul 30, 2021

Uh oh!

creydr commented Aug 2, 2021

Uh oh!

zhaozhanqi commented Aug 2, 2021

Uh oh!

creydr commented Aug 3, 2021

Uh oh!

abhat commented Aug 3, 2021

Uh oh!

abhat commented Aug 3, 2021

Uh oh!

openshift-ci bot commented Aug 3, 2021

Uh oh!

openshift-ci bot commented Aug 3, 2021

Uh oh!

openshift-bot commented Aug 4, 2021

Uh oh!

openshift-bot commented Aug 4, 2021

Uh oh!

openshift-ci bot commented Aug 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

creydr commented Jul 30, 2021 •

edited

Loading