Bug 1828427: Add CI test to check for critical alerts post upgrade success #24786

jottofar · 2020-03-27T18:28:45Z

No description provided.

jottofar · 2020-03-31T15:17:33Z

/override ci/prow/e2e-aws-fips

openshift-ci-robot · 2020-03-31T15:17:58Z

@jottofar: jottofar unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file.

Details

In response to this:

/override ci/prow/e2e-aws-fips

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jottofar · 2020-03-31T21:06:42Z

/test e2e-gcp-upgrade

test/e2e/upgrade/alert/alert.go

jottofar · 2020-04-01T12:42:07Z

/test e2e-gcp-upgrade

jottofar · 2020-04-01T20:29:03Z

/test e2e-gcp-upgrade

jottofar · 2020-04-02T13:13:52Z

/test e2e-gcp-upgrade

jottofar · 2020-04-02T14:51:52Z

/test e2e-gcp-upgrade

jottofar · 2020-04-02T21:04:59Z

/test e2e-gcp-upgrade

jottofar · 2020-04-02T21:10:18Z

/test e2e-gcp-upgrade

jottofar · 2020-04-02T21:13:54Z

/retest

jottofar · 2020-04-02T21:15:12Z

/test

openshift-ci-robot · 2020-04-27T17:16:29Z

@jottofar: This pull request references Bugzilla bug 1828427, which is invalid:

expected the bug to target the "4.5.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1828427: Add CI test to check for critical alerts post upgrade success

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jottofar · 2020-04-27T17:18:48Z

/bugzilla refresh

openshift-ci-robot · 2020-04-27T17:18:51Z

@jottofar: This pull request references Bugzilla bug 1828427, which is invalid:

expected the bug to target the "4.5.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jottofar · 2020-04-27T17:19:59Z

/bugzilla refresh

openshift-ci-robot · 2020-04-27T17:20:05Z

@jottofar: This pull request references Bugzilla bug 1828427, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-04-27T17:22:34Z

@jottofar: All pull requests linked via external trackers have merged: openshift/origin#24786. Bugzilla bug 1828427 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1828427: Add CI test to check for critical alerts post upgrade success

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

CI and nightly releases are not part of the official Red Hat Cincinnati graphs. This commit removes the channel property [1], which will result in the NoChannel condition [2], but will keep the CVO from attempting to find its current version in the official Cincinnati [3]. That in turn should keep the CVO from throwing a new, critical alert [4], which will keep it from running afoul of recent update e2e logic that forbids critical alerts [5]. [1]: https://github.com/openshift/installer/blob/4eca2efd615f8abd65f576721e2410b19f0d40d0/data/data/manifests/bootkube/cvo-overrides.yaml.template#L8 [2]: https://github.com/openshift/cluster-version-operator/blob/fa452c2d270f1f989f3868ef97ae8cf825713583/docs/user/status.md#nochannel [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1827378#c4 [4]: openshift/cluster-version-operator#357 [5]: openshift/origin#24786

wking · 2020-05-15T01:35:34Z

/cherrypick release-4.4

openshift-cherrypick-robot · 2020-05-15T01:38:31Z

@wking: #24786 failed to apply on top of branch "release-4.4":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	test/e2e/upgrade/upgrade.go
M	test/extended/prometheus/prometheus.go
M	test/extended/prometheus/prometheus_builds.go
M	test/extended/util/client.go
Falling back to patching base and 3-way merge...
Auto-merging test/extended/util/client.go
CONFLICT (content): Merge conflict in test/extended/util/client.go
Auto-merging test/extended/prometheus/prometheus_builds.go
CONFLICT (content): Merge conflict in test/extended/prometheus/prometheus_builds.go
Auto-merging test/extended/prometheus/prometheus.go
CONFLICT (content): Merge conflict in test/extended/prometheus/prometheus.go
Auto-merging test/e2e/upgrade/upgrade.go
Patch failed at 0001 Add CI test to check for crit alerts post upgrade

Details

In response to this:

/cherrypick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jottofar · 2020-06-24T20:30:09Z

/cherry-pick release-4.4

openshift-cherrypick-robot · 2020-06-24T20:33:20Z

@jottofar: #24786 failed to apply on top of branch "release-4.4":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	test/e2e/upgrade/upgrade.go
M	test/extended/prometheus/prometheus.go
M	test/extended/prometheus/prometheus_builds.go
M	test/extended/util/client.go
Falling back to patching base and 3-way merge...
Auto-merging test/extended/util/client.go
CONFLICT (content): Merge conflict in test/extended/util/client.go
Auto-merging test/extended/prometheus/prometheus_builds.go
CONFLICT (content): Merge conflict in test/extended/prometheus/prometheus_builds.go
Auto-merging test/extended/prometheus/prometheus.go
CONFLICT (content): Merge conflict in test/extended/prometheus/prometheus.go
Auto-merging test/e2e/upgrade/upgrade.go
Patch failed at 0001 Add CI test to check for crit alerts post upgrade

Details

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

We're doing better in updates now, and want to ratchet down to bar critical-alert noise during updates. The old 1m alertPeriodCheckMinutes landed with this test in 3b8cb3c (Add CI test to check for crit alerts post upgrade, 2020-03-27, openshift#24786). DurationSinceStartInSeconds, which I'm using now, landed in ace1345 (test: Allow tests that check invariants over time to be constrained, 2021-01-06, openshift#25784).

CI and nightly releases are not part of the official Red Hat Cincinnati graphs. This commit removes the channel property [1], which will result in the NoChannel condition [2], but will keep the CVO from attempting to find its current version in the official Cincinnati [3]. That in turn should keep the CVO from throwing a new, critical alert [4], which will keep it from running afoul of recent update e2e logic that forbids critical alerts [5]. [1]: https://github.com/openshift/installer/blob/4eca2efd615f8abd65f576721e2410b19f0d40d0/data/data/manifests/bootkube/cvo-overrides.yaml.template#L8 [2]: https://github.com/openshift/cluster-version-operator/blob/fa452c2d270f1f989f3868ef97ae8cf825713583/docs/user/status.md#nochannel [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1827378#c4 [4]: openshift/cluster-version-operator#357 [5]: openshift/origin#24786

CI and nightly releases are not part of the official Red Hat Cincinnati graphs. This commit removes the channel property [1], which will result in the NoChannel condition [2], but will keep the CVO from attempting to find its current version in the official Cincinnati [3]. That in turn should keep the CVO from throwing a new, critical alert [4], which will keep it from running afoul of recent update e2e logic that forbids critical alerts [5]. See this related PR [6] which does this for other platforms IPI install step. [1]: https://github.com/openshift/installer/blob/4eca2efd615f8abd65f576721e2410b19f0d40d0/data/data/manifests/bootkube/cvo-overrides.yaml.template#L8 [2]: https://github.com/openshift/cluster-version-operator/blob/fa452c2d270f1f989f3868ef97ae8cf825713583/docs/user/status.md#nochannel [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1827378#c4 [4]: openshift/cluster-version-operator#357 [5]: openshift/origin#24786 [6]: openshift/release#8631

CI and nightly releases are not part of the official Red Hat Cincinnati graphs. This commit removes the channel property [1], which will result in the NoChannel condition [2], but will keep the CVO from attempting to find its current version in the official Cincinnati [3]. That in turn should keep the CVO from throwing a new, critical alert [4], which will keep it from running afoul of recent update e2e logic that forbids critical alerts [5]. See this related PR [6] which does this for other platforms IPI install step. [1]: https://github.com/openshift/installer/blob/4eca2efd615f8abd65f576721e2410b19f0d40d0/data/data/manifests/bootkube/cvo-overrides.yaml.template#L8 [2]: https://github.com/openshift/cluster-version-operator/blob/fa452c2d270f1f989f3868ef97ae8cf825713583/docs/user/status.md#nochannel [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1827378#c4 [4]: openshift/cluster-version-operator#357 [5]: openshift/origin#24786 [6]: openshift/release#8631 Co-authored-by: W. Trevor King <[email protected]>

CI and nightly releases are not part of the official Red Hat Cincinnati graphs. This commit removes the channel property [1], which will result in the NoChannel condition [2], but will keep the CVO from attempting to find its current version in the official Cincinnati [3]. That in turn should keep the CVO from throwing a new, critical alert [4], which will keep it from running afoul of recent update e2e logic that forbids critical alerts [5]. See this related PR [6] which does this for other platforms IPI install step. [1]: https://github.com/openshift/installer/blob/4eca2efd615f8abd65f576721e2410b19f0d40d0/data/data/manifests/bootkube/cvo-overrides.yaml.template#L8 [2]: https://github.com/openshift/cluster-version-operator/blob/fa452c2d270f1f989f3868ef97ae8cf825713583/docs/user/status.md#nochannel [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1827378#c4 [4]: openshift/cluster-version-operator#357 [5]: openshift/origin#24786 [6]: openshift/release#8631 Co-authored-by: W. Trevor King <[email protected]> Co-authored-by: W. Trevor King <[email protected]>

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 27, 2020

openshift-ci-robot requested review from deads2k and soltysh March 27, 2020 18:32

jottofar force-pushed the ota-141-add-ci-alerts branch 7 times, most recently from 2cef71b to 7463695 Compare March 31, 2020 15:11

jottofar force-pushed the ota-141-add-ci-alerts branch from 7463695 to 48982f5 Compare March 31, 2020 18:41

jottofar force-pushed the ota-141-add-ci-alerts branch from 48982f5 to 5ded23f Compare March 31, 2020 23:31

p0lyn0mial reviewed Apr 1, 2020

View reviewed changes

test/e2e/upgrade/alert/alert.go Show resolved Hide resolved

jottofar force-pushed the ota-141-add-ci-alerts branch 2 times, most recently from ef7591c to e742806 Compare April 1, 2020 17:26

jottofar force-pushed the ota-141-add-ci-alerts branch from e742806 to 7b8a966 Compare April 1, 2020 22:32

jottofar force-pushed the ota-141-add-ci-alerts branch from 7b8a966 to 13f2225 Compare April 2, 2020 19:02

jottofar force-pushed the ota-141-add-ci-alerts branch 2 times, most recently from c609289 to 44a59e6 Compare April 3, 2020 12:23

jottofar changed the title ~~Add CI test to check for critical alerts post upgrade success~~ Bug 1828427: Add CI test to check for critical alerts post upgrade success Apr 27, 2020

openshift-ci-robot added the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Apr 27, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Apr 27, 2020

openshift-ci-robot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. and removed bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. labels Apr 27, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Apr 27, 2020

openshift-merge-robot merged commit 3be2bc2 into openshift:master Apr 27, 2020

wking mentioned this pull request Apr 29, 2020

ci-operator/step-registry/ipi/install: Remove ClusterVersion channel openshift/release#8631

Merged

wking mentioned this pull request Jun 15, 2020

test/extended/prometheus: Add test for no alerts in pending state #25112

Closed

sttts mentioned this pull request Sep 14, 2020

Bug 1878862: Revert "Ignore KubeAPIErrorBudgetBurn alert" #25500

Merged

This was referenced Feb 22, 2021

test/e2e/upgrade/alert: Extend to testDuration, not just 1m #25912

Closed

test: Require no alerts during upgrades #25904

Merged

Bug 1828427: Add CI test to check for critical alerts post upgrade success #24786

Bug 1828427: Add CI test to check for critical alerts post upgrade success #24786

Uh oh!

Conversation

jottofar commented Mar 27, 2020

Uh oh!

jottofar commented Mar 31, 2020

Uh oh!

openshift-ci-robot commented Mar 31, 2020

Uh oh!

jottofar commented Mar 31, 2020

Uh oh!

Uh oh!

jottofar commented Apr 1, 2020

Uh oh!

jottofar commented Apr 1, 2020

Uh oh!

jottofar commented Apr 2, 2020

Uh oh!

jottofar commented Apr 2, 2020

Uh oh!

jottofar commented Apr 2, 2020

Uh oh!

jottofar commented Apr 2, 2020

Uh oh!

jottofar commented Apr 2, 2020

Uh oh!

jottofar commented Apr 2, 2020

Uh oh!

openshift-ci-robot commented Apr 27, 2020

Uh oh!

jottofar commented Apr 27, 2020

Uh oh!

openshift-ci-robot commented Apr 27, 2020

Uh oh!

jottofar commented Apr 27, 2020

Uh oh!

openshift-ci-robot commented Apr 27, 2020

Uh oh!

openshift-ci-robot commented Apr 27, 2020

Uh oh!

wking commented May 15, 2020

Uh oh!

openshift-cherrypick-robot commented May 15, 2020

Uh oh!

jottofar commented Jun 24, 2020

Uh oh!

openshift-cherrypick-robot commented Jun 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants