Bug 2009879: Add admin ack Upgradeable condition gate test #26446

jottofar · 2021-09-02T16:21:37Z

Since only later 4.8.z releases will contain the admin ack functionality, the test first checks for presence of the admin-gates configmap. If not found, indicating that the admin ack functionality is not part the baseline being tested, the test simply returns successfully.
Otherwise, test first verifies that Upgradeable condition is false for correct reason and with correct message. It then modifies the admin-acks configmap to ack the necessary admin ack gate and then waits for the Upgradeable condition to change to true.

openshift-ci · 2021-09-02T16:21:44Z

@jottofar: This pull request references Bugzilla bug 1999092, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1978376 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1978376 targets the "4.9.0" release, which is one of the valid target releases: 4.9.0
bug has dependents

Requesting review from QA contact:
/cc @jianlinliu

Details

In response to this:

WIP: Bug 1999092: Add admin ack Upgradeable condition gate test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-09-02T21:16:14Z

@jottofar: This pull request references Bugzilla bug 1999092, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1978376 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1978376 targets the "4.9.0" release, which is one of the valid target releases: 4.9.0
bug has dependents

Requesting review from QA contact:
/cc @jianlinliu

Details

In response to this:

WIP: Bug 1999092: Add admin ack Upgradeable condition gate test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-09-02T21:19:04Z

@jottofar: This pull request references Bugzilla bug 1999092, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1978376 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1978376 targets the "4.9.0" release, which is one of the valid target releases: 4.9.0
bug has dependents

Requesting review from QA contact:
/cc @jianlinliu

Details

In response to this:

WIP: Bug 1999092: Add admin ack Upgradeable condition gate test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

test/e2e/upgrade/adminack/adminack.go

jottofar · 2021-09-17T15:15:05Z

/retest

…e-1.22-api-removals-in-4.9 Avoid [1]: alert ClusterNotUpgradeable fired for 1115 seconds with labels: {condition="Upgradeable", endpoint="metrics", name="version", severity="warning"} for tests that install 4.8 and then sit on it for over an hour, now that [2] has landed in 4.8. [3] will follow up with code to handle CI that installs an earlier release and updates into 4.8 or later. [1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-serial/1443018123379740672 [2]: openshift/cluster-version-operator#647 [3]: openshift/origin#26446

test/e2e/upgrade/adminack/adminack.go

jottofar · 2021-09-29T21:36:35Z

/retitle Bug 1999092: Add admin ack Upgradeable condition gate test

test/e2e/upgrade/adminack/adminack.go

wking · 2021-09-29T21:53:41Z

test/e2e/upgrade/upgrade.go

 // AllTests includes all tests (minimal + disruption)
 func AllTests() []upgrades.Test {
 	return []upgrades.Test{
+		&adminack.UpgradeTest{},


I don't know if the order here is important, but I'd like this test to run at the end of the suite (I'd have expected to have that happen via a [Late] annotation). But the alert.UpgradeTest{} precedent makes it seem like order does matter here. I'm not entirely sure how these are positioned relative to the update itself, but the logging makes them seem like they're currently running early in the job:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/26446/pull-ci-openshift-origin-release-4.8-e2e-aws-upgrade/1438885249361645568/artifacts/e2e-aws-upgrade/openshift-e2e-test/artifacts/e2e.log | grep -n 'Cluster should remain functional during upgrade\|admin ack' | grep -B100 passed 84:started: (0/5/5) "[sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]" 95:[It] Cluster should remain functional during upgrade [Disruptive] [Serial] 167:STEP: Setting up admin ack test 170:Sep 17 16:10:42.260: INFO: Skipping admin ack test. Admin ack is not in this baseline. 1751:passed: (2h47m1s) 2021-09-17T18:57:41 "[sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]"

(with line 170 presumably being written quite a while before line 1751). Running later (post update) also protects 4.7 -> 4.8 jobs from having Upgradeable=False alerting go off, because the openshift/release#22368 approach will not protect those jobs (and some jobs run slow parallel-conformance suites after completing the update).

It is order dependent. I put it early since the the upgrade cannot proceed until the ack is in place and the test ends leaving the ack in place.

wking · 2021-10-01T20:36:34Z

/retitle Bug 2009879: Add admin ack Upgradeable condition gate test

No target on bug 2009879 yet, while we work out whether this PR will be 4.8-only or not, but pointing away from the old bug series so the Bugzilla bot doesn't attempt to re-attach this PR.

jottofar · 2021-10-06T20:11:21Z

/test verify

jottofar · 2021-10-11T13:40:14Z

/retest

jottofar · 2021-10-12T13:42:25Z

/hold
PR will dup'ed to master and then back-ported to 4.8. Continuing with this PR for test purposes only.

jottofar · 2021-10-13T13:11:40Z

/test verify

openshift-ci · 2021-10-13T17:49:08Z

@jottofar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-upgrade	`08db7f9`	link	true	`/test e2e-gcp-upgrade`
ci/prow/e2e-metal-ipi-ovn-ipv6	`08db7f9`	link	true	`/test e2e-metal-ipi-ovn-ipv6`
ci/prow/e2e-aws-upgrade	`08db7f9`	link	true	`/test e2e-aws-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

jianlinliu · 2021-11-08T07:48:30Z

/lgtm

openshift-ci · 2021-11-08T07:51:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jianlinliu, jottofar
To complete the pull request process, please assign deads2k after the PR has been reviewed.
You can assign the PR to them by writing /assign @deads2k in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wking · 2021-11-22T20:14:47Z

Replaced by #26635, backporting the generic logic that landed in master and 4.9.

/close

openshift-ci · 2021-11-22T20:18:23Z

@wking: Closed this PR.

Details

In response to this:

Replaced by #26635, backporting the generic logic that landed in master and 4.9.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. labels Sep 2, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Sep 2, 2021

openshift-ci bot requested review from deads2k, jianlinliu and smarterclayton September 2, 2021 16:21

jottofar force-pushed the bug-1999092 branch from c3107bc to e62599d Compare September 2, 2021 21:11

jottofar force-pushed the bug-1999092 branch from e62599d to b249623 Compare September 2, 2021 21:22

jianlinliu reviewed Sep 16, 2021

View reviewed changes

test/e2e/upgrade/adminack/adminack.go Outdated Show resolved Hide resolved

jottofar force-pushed the bug-1999092 branch from b249623 to 360adde Compare September 16, 2021 20:53

openshift-ci bot mentioned this pull request Sep 28, 2021

Bug 1999092: Add and enable admin ack Upgradeable condition gate openshift/cluster-version-operator#647

Merged

wking mentioned this pull request Sep 29, 2021

ci-operator/step-registry/openshift/e2e/test: Best-effort ack-4.8-kube-1.22-api-removals-in-4.9 openshift/release#22368

Merged

wking reviewed Sep 29, 2021

View reviewed changes

test/e2e/upgrade/adminack/adminack.go Outdated Show resolved Hide resolved

wking reviewed Sep 29, 2021

View reviewed changes

test/e2e/upgrade/adminack/adminack.go Outdated Show resolved Hide resolved

openshift-ci bot changed the title ~~WIP: Bug 1999092: Add admin ack Upgradeable condition gate test~~ Bug 1999092: Add admin ack Upgradeable condition gate test Sep 29, 2021

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 29, 2021

wking reviewed Sep 29, 2021

View reviewed changes

test/e2e/upgrade/adminack/adminack.go Outdated Show resolved Hide resolved

wking reviewed Sep 29, 2021

View reviewed changes

jottofar force-pushed the bug-1999092 branch from 360adde to 4e51920 Compare September 30, 2021 17:55

openshift-ci bot changed the title ~~Bug 1999092: Add admin ack Upgradeable condition gate test~~ Bug 2009879: Add admin ack Upgradeable condition gate test Oct 1, 2021

jottofar force-pushed the bug-1999092 branch 2 times, most recently from eab2736 to 2cdcbcc Compare October 5, 2021 21:06

jottofar force-pushed the bug-1999092 branch from 2cdcbcc to 5809eb0 Compare October 6, 2021 19:28

jottofar force-pushed the bug-1999092 branch 3 times, most recently from 52fe0bd to 513c7da Compare October 7, 2021 00:49

jottofar force-pushed the bug-1999092 branch 6 times, most recently from 2729e06 to 6b175c4 Compare October 12, 2021 13:41

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 12, 2021

jottofar force-pushed the bug-1999092 branch 2 times, most recently from 98091aa to b4ce478 Compare October 12, 2021 20:48

test/e2e/upgrade/adminack: Add admin ack test

08db7f9

jottofar force-pushed the bug-1999092 branch from b4ce478 to 08db7f9 Compare October 13, 2021 13:43

openshift-ci bot assigned jianlinliu Nov 8, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2021

openshift-ci bot closed this Nov 22, 2021

Bug 2009879: Add admin ack Upgradeable condition gate test #26446

Bug 2009879: Add admin ack Upgradeable condition gate test #26446

Uh oh!

Conversation

jottofar commented Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Sep 2, 2021

Uh oh!

openshift-ci bot commented Sep 2, 2021

Uh oh!

openshift-ci bot commented Sep 2, 2021

Uh oh!

Uh oh!

jottofar commented Sep 17, 2021

Uh oh!

Uh oh!

Uh oh!

jottofar commented Sep 29, 2021

Uh oh!

Uh oh!

wking Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

jottofar Oct 12, 2021

Choose a reason for hiding this comment

Uh oh!

wking commented Oct 1, 2021

Uh oh!

jottofar commented Oct 6, 2021

Uh oh!

jottofar commented Oct 11, 2021

Uh oh!

jottofar commented Oct 12, 2021

Uh oh!

jottofar commented Oct 13, 2021

Uh oh!

openshift-ci bot commented Oct 13, 2021

Uh oh!

jianlinliu commented Nov 8, 2021

Uh oh!

openshift-ci bot commented Nov 8, 2021

Uh oh!

wking commented Nov 22, 2021

Uh oh!

openshift-ci bot commented Nov 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jottofar commented Sep 2, 2021 •

edited

Loading