Bug 2010365: OpenShift Alerting Rules Style-Guide Compliance #800

DavidHurta · 2022-07-21T14:30:03Z

Add missing namespace labels openshift-cluster-version to alerting rules to comply with
the style guidance. Alerts should include a namespace label indicating the alert's source.

openshift-ci · 2022-07-21T14:30:19Z

@Davoska: This pull request references Bugzilla bug 2010365, which is invalid:

expected the bug to target the "4.12.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 2010365: OpenShift Alerting Rules Style-Guide Compliance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-07-21T14:31:41Z

@Davoska: This pull request references Bugzilla bug 2010365, which is invalid:

expected the bug to target the "4.12.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 2010365: OpenShift Alerting Rules Style-Guide Compliance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

DavidHurta · 2022-07-21T14:37:05Z

/bugzilla refresh

openshift-ci · 2022-07-21T14:37:16Z

@Davoska: This pull request references Bugzilla bug 2010365, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.12.0) matches configured target release for branch (4.12.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @jianlinliu

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

DavidHurta · 2022-07-22T06:25:31Z

/retest-required

install/0000_90_cluster-version-operator_02_servicemonitor.yaml

wking · 2022-07-25T05:42:03Z

install/0000_90_cluster-version-operator_02_servicemonitor.yaml

        cluster_operator_up{job="cluster-version-operator"} == 0
      for: 10m
      labels:
+        namespace: openshift-cluster-version


ClusterOperatorDown already has namespace in Telemetry (this time via cluster_operator_up), so I'd rather drop this line from the commit as redundant.

We could consider a max by (namespace, name, ...) (...) aggregation, if we wanted to exclude pod to avoid alert churn whenever the CVO pod cycled. But we're not all that consistent on this front today, so if we do decide to do this sort of thing, we should probably put in enough thought to make the change consistently (e.g. I'm not sure why we are including endpoint but not service in ClusterNotUpgradeable today).

Thank you, fixed in d525eb9.

I have also added the aggregation operator max by to CannotRetrieveUpdates, ClusterOperatorDown, ClusterOperatorDegraded, and ClusterOperatorFlapping to avoid the alert churn whenever the CVO pod cycles in the new commit 2b8ef2a.

I am not sure of the purpose of the endpoint or potentially service in ClusterNotUpgradeable, so I have not made this modification.

install/0000_90_cluster-version-operator_02_servicemonitor.yaml

DavidHurta · 2022-07-28T08:08:15Z

/hold addressing comments

…pace labels to alerts Add missing namespace labels to alerting rules to comply with the style guidance [1]. Alerts should include a namespace label indicating the alert's source. Either add the static label or modify the PromQL expression to include the namespace label. [1] https://github.com/openshift/enhancements/blob/master/enhancements/monitoring/alerting-consistency.md#style-guide

DavidHurta · 2022-08-01T12:04:32Z

Thank you, Trevor, for the explanations. It seems that I forgot about the purpose of aggregation operators and receiving labels implicitly from metrics 🤦. I wanted to be explicit about the alerts having the namespace. However, I now understand that it's redundant.

Fixed the redundant labels namespace in the new commit d525eb9.

Added aggregation operators to some alerts to exclude necessary labels (mainly for the purpose of avoiding multiple alerts when the CVO pod cycles) in the new commit 2b8ef2a.

/unhold

DavidHurta · 2022-08-01T17:04:45Z

/hold
I'll double check the use of aggregation operators in 2b8ef2a.

DavidHurta · 2022-08-01T18:00:58Z

2b8ef2a -> ffb2602 wording of the commit message.

/unhold

openshift-ci · 2022-08-05T20:10:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Davoska, LalatenduMohanty, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [LalatenduMohanty,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2022-08-05T20:23:40Z

/retest-required

Remaining retests: 2 against base HEAD 0f8c533 and 8 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-06T01:07:35Z

/retest-required

Remaining retests: 1 against base HEAD 0f8c533 and 7 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-06T06:04:10Z

/retest-required

Remaining retests: 0 against base HEAD 0f8c533 and 6 for PR HEAD 66fc016 in total

DavidHurta · 2022-08-08T07:30:48Z

/retest

openshift-ci-robot · 2022-08-08T13:01:48Z

/retest-required

Remaining retests: 2 against base HEAD 02838c7 and 5 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-08T17:35:32Z

/retest-required

Remaining retests: 1 against base HEAD 02838c7 and 4 for PR HEAD 66fc016 in total

wking · 2022-08-08T18:09:01Z

This run passed, except for timing out during teardown.

/override ci/prow/e2e-agnostic-upgrade

openshift-ci · 2022-08-08T18:12:42Z

@wking: Overrode contexts on behalf of wking: ci/prow/e2e-agnostic-upgrade

Details

In response to this:

This run passed, except for timing out during teardown.

/override ci/prow/e2e-agnostic-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2022-08-08T21:15:20Z

/retest-required

Remaining retests: 0 against base HEAD 02838c7 and 3 for PR HEAD 66fc016 in total

DavidHurta · 2022-08-11T08:32:47Z

@wking, I guess we need to run the /override again? 🤔 I am not sure why it's still not passing because there weren't any merges to the master branch since that.

openshift-ci-robot · 2022-08-11T16:04:41Z

/retest-required

Remaining retests: 2 against base HEAD c699d55 and 2 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-11T20:12:29Z

/retest-required

Remaining retests: 1 against base HEAD c699d55 and 1 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-12T00:17:11Z

/retest-required

Remaining retests: 0 against base HEAD c699d55 and 0 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-12T04:05:22Z

/hold

Revision 66fc016 was retested 9 times: holding

DavidHurta · 2022-08-16T08:35:03Z

#800 (comment):

This run passed, except for timing out during teardown.

/override ci/prow/e2e-agnostic-upgrade

Unholding the PR and retesting, so we can potentially override the ci/prow/e2e-agnostic-upgrade again, because the head of the master branch has changed.

DavidHurta · 2022-08-16T08:39:12Z

/unhold

DavidHurta · 2022-08-16T08:40:40Z

/retest

openshift-ci-robot · 2022-08-16T12:32:27Z

/retest-required

Remaining retests: 2 against base HEAD 348add6 and 8 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-16T16:25:07Z

/retest-required

Remaining retests: 2 against base HEAD bfa3017 and 7 for PR HEAD 66fc016 in total

wking · 2022-08-16T17:12:29Z

Still hitting orthogonal disruption failures because the origin suite doesn't scale disruption for our A->B->A rollback presubmit.

/override ci/prow/e2e-agnostic-upgrade

openshift-ci · 2022-08-16T17:13:28Z

@wking: Overrode contexts on behalf of wking: ci/prow/e2e-agnostic-upgrade

Details

In response to this:

Still hitting orthogonal disruption failures because the origin suite doesn't scale disruption for our A->B->A rollback presubmit.

/override ci/prow/e2e-agnostic-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2022-08-16T17:25:52Z

/retest-required

Remaining retests: 1 against base HEAD bfa3017 and 6 for PR HEAD 66fc016 in total

openshift-ci-robot · 2022-08-16T17:49:03Z

/retest-required

Remaining retests: 0 against base HEAD bfa3017 and 5 for PR HEAD 66fc016 in total

openshift-ci · 2022-08-16T19:50:54Z

@Davoska: All pull requests linked via external trackers have merged:

openshift/cluster-version-operator#800

Bugzilla bug 2010365 has been moved to the MODIFIED state.

Details

In response to this:

Bug 2010365: OpenShift Alerting Rules Style-Guide Compliance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-08-16T19:51:13Z

@Davoska: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci bot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jul 21, 2022

openshift-ci bot requested review from jottofar and wking July 21, 2022 14:32

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jul 21, 2022

openshift-ci bot requested a review from jianlinliu July 21, 2022 14:37

wking reviewed Jul 25, 2022

View reviewed changes

install/0000_90_cluster-version-operator_02_servicemonitor.yaml Outdated Show resolved Hide resolved

wking reviewed Jul 25, 2022

View reviewed changes

install/0000_90_cluster-version-operator_02_servicemonitor.yaml Outdated Show resolved Hide resolved

wking reviewed Jul 25, 2022

View reviewed changes

install/0000_90_cluster-version-operator_02_servicemonitor.yaml Outdated Show resolved Hide resolved

wking reviewed Jul 25, 2022

View reviewed changes

install/0000_90_cluster-version-operator_02_servicemonitor.yaml Outdated Show resolved Hide resolved

wking reviewed Jul 25, 2022

View reviewed changes

install/0000_90_cluster-version-operator_02_servicemonitor.yaml Outdated Show resolved Hide resolved

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 28, 2022

DavidHurta force-pushed the bug-2010365-openshift-alerting-rules-compliance branch from 900a462 to 2b8ef2a Compare August 1, 2022 11:32

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 1, 2022

DavidHurta requested a review from wking August 1, 2022 12:07

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 1, 2022

DavidHurta force-pushed the bug-2010365-openshift-alerting-rules-compliance branch from 2b8ef2a to ffb2602 Compare August 1, 2022 17:59

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 1, 2022

openshift-ci bot assigned wking Aug 5, 2022

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 12, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 16, 2022

openshift-merge-robot merged commit 2d7057c into openshift:master Aug 16, 2022

Bug 2010365: OpenShift Alerting Rules Style-Guide Compliance #800

Bug 2010365: OpenShift Alerting Rules Style-Guide Compliance #800

Uh oh!

Conversation

DavidHurta commented Jul 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Jul 21, 2022

Uh oh!

openshift-ci bot commented Jul 21, 2022

Uh oh!

DavidHurta commented Jul 21, 2022

Uh oh!

openshift-ci bot commented Jul 21, 2022

Uh oh!

DavidHurta commented Jul 22, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wking Jul 25, 2022

Choose a reason for hiding this comment

Uh oh!

DavidHurta Aug 1, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DavidHurta commented Jul 28, 2022

Uh oh!

DavidHurta commented Aug 1, 2022

Uh oh!

DavidHurta commented Aug 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHurta commented Aug 1, 2022

Uh oh!

openshift-ci bot commented Aug 5, 2022

Uh oh!

openshift-ci-robot commented Aug 5, 2022

Uh oh!

openshift-ci-robot commented Aug 6, 2022

Uh oh!

openshift-ci-robot commented Aug 6, 2022

Uh oh!

DavidHurta commented Aug 8, 2022

Uh oh!

openshift-ci-robot commented Aug 8, 2022

Uh oh!

openshift-ci-robot commented Aug 8, 2022

Uh oh!

wking commented Aug 8, 2022

Uh oh!

openshift-ci bot commented Aug 8, 2022

Uh oh!

openshift-ci-robot commented Aug 8, 2022

Uh oh!

DavidHurta commented Aug 11, 2022

Uh oh!

openshift-ci-robot commented Aug 11, 2022

Uh oh!

openshift-ci-robot commented Aug 11, 2022

Uh oh!

openshift-ci-robot commented Aug 12, 2022

Uh oh!

openshift-ci-robot commented Aug 12, 2022

Uh oh!

DavidHurta commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHurta commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHurta commented Aug 16, 2022

Uh oh!

openshift-ci-robot commented Aug 16, 2022

Uh oh!

openshift-ci-robot commented Aug 16, 2022

Uh oh!

wking commented Aug 16, 2022

DavidHurta commented Jul 21, 2022 •

edited

Loading

DavidHurta commented Aug 1, 2022 •

edited

Loading

DavidHurta commented Aug 16, 2022 •

edited

Loading

DavidHurta commented Aug 16, 2022 •

edited

Loading