Bug 1970975: upgrade test: expect upgrades to take longer on AWS #26219

vrutkovs · 2021-06-10T17:25:12Z

Due to AWS LB issue apiserver rollout takes twice as much. This bumps upgrade duration on AWS to 90 mins.

Upgrade tests:

test upgrade 4.7 https://github.com/openshift/origin/pull/26219 aws

[sig-cluster-lifecycle] cluster upgrade should complete in 60m (90m on AWS) | 1h7m50s

test upgrade 4.8.0-fc.8 https://github.com/openshift/origin/pull/26219 aws

[sig-cluster-lifecycle] cluster upgrade should complete in 60m (90m on AWS)&nbsp;expand_less | 1h32m30s
-- | --
upgrade to registry.build01.ci.openshift.org/ci-ln-chzn6vb/release@sha256:1e6a8003e66a4b68e03128e204b679bd6db5ca354edb802f0709699b0490cde8 took too long: 92.50404250443333

bparees · 2021-06-10T17:27:26Z

test/e2e/upgrade/upgrade.go

+	infra, err := c.ConfigV1().Infrastructures().Get(context.Background(), "cluster", metav1.GetOptions{})
+	framework.ExpectNoError(err)
+	if infra.Status.PlatformStatus.Type == configv1.AWSPlatformType {
+		// due to AWS LB bug upgrades take longer on AWS


is this a bug in our AWS LB code, or in AWS itself?

are we tracking a resolution? this would also impact customers, right?

(in that their upgrades will take longer. obviously they don't have the same timeout to worry about)

A bug in AWS NLB healthchecks, which we worked around by making kubeapi-operator be resilient. This makes us wait twice - ~12m during reboot and ~12m during apiserver rollout.

IIUC its this issue, but @smarterclayton might have better reference for this.

/cc @smarterclayton

It's not the nu26 issue. It's https://bugzilla.redhat.com/show_bug.cgi?id=1943804. As a result, we take 12 extra minutes to rollout apiserver twice (120s -> 240s per machine, 3 machines, 2 separate rollouts, machines and apiserver)

Go ahead and change the JUnit to be "Cluster upgrade should complete within 60m (90m on AWS)"

Done. Added bug reference to the comment

bparees · 2021-06-10T17:32:46Z

/lgtm

note, this might push us over the timeout limits on the multistep upgrade tests in the future (e.g. 4.8->4.9->4.10)

vrutkovs · 2021-06-10T17:33:33Z

note, this might push us over the timeout limits on the multistep upgrade tests in the future (e.g. 4.8->4.9->4.10)

Right, we'll work with DPTP to extend those

wking · 2021-06-10T18:13:12Z

There is also #26202 in this space with a blanket raise to 90m, vs. this PR's platform-specific raise.

openshift-bot · 2021-06-10T19:15:13Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T19:39:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T19:51:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T20:15:33Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T21:03:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T21:15:03Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T21:53:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T22:04:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T22:40:00Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T23:04:03Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T23:28:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-10T23:52:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T00:04:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T01:07:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T01:33:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T01:46:00Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T03:17:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T05:01:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T06:19:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

Due to AWS LB issue apiserver rollout takes twice as much. This bumps upgrade duration on AWS to 90 mins

openshift-ci · 2021-06-11T12:00:58Z

@vrutkovs: This pull request references Bugzilla bug 1970315, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1970315: upgrade test: expect upgrades to take longer on AWS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2021-06-11T12:01:40Z

/bugzilla refresh

openshift-ci · 2021-06-11T12:01:43Z

@vrutkovs: This pull request references Bugzilla bug 1970315, which is invalid:

expected the bug to target the "4.8.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2021-06-11T12:04:31Z

/bugzilla refresh

openshift-ci · 2021-06-11T12:04:42Z

@vrutkovs: This pull request references Bugzilla bug 1970315, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.0) matches configured target release for branch (4.8.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-06-11T12:05:03Z

@vrutkovs: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

upgrade test: expect upgrades to take longer on AWS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stbenjam · 2021-06-11T13:05:40Z

@vrutkovs Do you have a new bug to use for this PR? Looks like the other one which you removed was the wrong bug.

/lgtm

openshift-ci · 2021-06-11T13:05:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, stbenjam, vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/OWNERS~~ [bparees]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-06-11T13:10:04Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-11T13:22:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-06-11T13:46:34Z

@vrutkovs: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-disruptive	`587d519`	link	`/test e2e-aws-disruptive`
ci/prow/e2e-gcp-disruptive	`587d519`	link	`/test e2e-gcp-disruptive`
ci/prow/e2e-aws-upgrade	`587d519`	link	`/test e2e-aws-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-06-11T13:59:22Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-06-11T15:15:03Z

@vrutkovs: This pull request references Bugzilla bug 1970975, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.0) matches configured target release for branch (4.8.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @wangke19

Details

In response to this:

Bug 1970975: upgrade test: expect upgrades to take longer on AWS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-06-11T15:16:21Z

@vrutkovs: All pull requests linked via external trackers have merged:

openshift/origin#26219

Bugzilla bug 1970975 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1970975: upgrade test: expect upgrades to take longer on AWS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from bparees and deads2k June 10, 2021 17:25

bparees reviewed Jun 10, 2021

View reviewed changes

openshift-ci bot requested a review from smarterclayton June 10, 2021 17:30

openshift-ci bot assigned bparees Jun 10, 2021

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jun 10, 2021

upgrade test: expect upgrades to take longer on AWS

587d519

Due to AWS LB issue apiserver rollout takes twice as much. This bumps upgrade duration on AWS to 90 mins

openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot requested a review from zhaozhanqi June 11, 2021 12:04

vrutkovs changed the title ~~Bug 1970315: upgrade test: expect upgrades to take longer on AWS~~ upgrade test: expect upgrades to take longer on AWS Jun 11, 2021

openshift-ci bot removed the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot removed the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 11, 2021

openshift-ci bot assigned stbenjam Jun 11, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2021

vrutkovs changed the title ~~upgrade test: expect upgrades to take longer on AWS~~ Bug 1970975: upgrade test: expect upgrades to take longer on AWS Jun 11, 2021

openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jun 11, 2021

openshift-ci bot requested a review from wangke19 June 11, 2021 15:15

openshift-merge-robot merged commit 15ba47a into openshift:master Jun 11, 2021

openshift-ci bot mentioned this pull request Jul 13, 2021

Bug 1970975: upgrade: extend upgrade duration to 105mins on AWS #26252

Merged

DavidHurta mentioned this pull request Mar 3, 2022

Bug 1885322: Increase OVN upgrade timeout by 15m #26878

Closed

Bug 1970975: upgrade test: expect upgrades to take longer on AWS #26219

Bug 1970975: upgrade test: expect upgrades to take longer on AWS #26219

Uh oh!

Conversation

vrutkovs commented Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bparees Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

bparees Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

vrutkovs Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

smarterclayton Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

smarterclayton Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

vrutkovs Jun 11, 2021

Choose a reason for hiding this comment

Uh oh!

bparees commented Jun 10, 2021

Uh oh!

vrutkovs commented Jun 10, 2021

Uh oh!

wking commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 10, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-bot commented Jun 11, 2021

Uh oh!

openshift-ci bot commented Jun 11, 2021

Uh oh!

vrutkovs commented Jun 11, 2021

Uh oh!

openshift-ci bot commented Jun 11, 2021

Uh oh!

vrutkovs commented Jun 11, 2021

Uh oh!

openshift-ci bot commented Jun 11, 2021

Uh oh!

openshift-ci bot commented Jun 11, 2021

vrutkovs commented Jun 10, 2021 •

edited

Loading