Adjust alert severities

[michaelgugino]

The alerts that are currently "critical" are not actually
situations that jeopardize the cluster's immediate health
or ability to run workloads. No one needs to be paged
in the middle of the night for these alerts.

This commit reduces severity to warning to reflect
degraded state.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: michaelgugino
To complete the pull request process, please assign jottofar after the PR has been reviewed.
You can assign the PR to them by writing /assign @jottofar in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sdodson]

/hold
I don't disagree with this, but in order to reduce churn I'd like to see openshift/enhancements#637 accepted first

[wking]

I'm not really interested in getting in the middle here. @smarterclayton was pushing reasonably firmly for critical back when these all landed. I'll let folks fight out what they think the severity level should be, and focus on trying to reduce the number of times any of these go off ;).

[smarterclayton]

As per discussions in the alert enhancement, and in slack, and a general "when is available and degraded important" - degraded generally means 'still working but not making progress / unable to give full status'. By that definition degraded =~ warning, and timing should be "in the morning".

By that measure available=false = critical, I am ok (if trevor is) with splitting ClusterOperatorDown to only include available and reachable and be critical, then to make ClusterOperatorDegraded cover the degraded condition and have a longer for.

Finally, if we do that, suppressing operation condition alerts until install is complete (until the operator reaches level) may also be useful.

[michaelgugino]

These CVO conditions clearly don't meet the criteria of 'get out of bed in the middle of the night.' Workloads will continue to run, new workloads will continue to be scheduled, existing operators will continue to run. The CVO being completely down could persist indefinitely with no adverse impact on workloads.

[wking]

... ClusterOperatorDegraded cover the degraded condition and have a longer for.

I've spun this portion out into #554. I'm more interested in the longer for, but I also dropped the severity to warning. Let me know if you'd rather me leave the severity changing to this PR or not. I haven't addressed ClusterOperatorDown's timing yet, but I'll try to get something in place for that soon, and then folks can tell me if they want me to twiddle severity for it in that PR or leave that to this PR as well.

[openshift-ci]

@michaelgugino: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-agnostic-upgrade	5752ec3	link	/test e2e-agnostic-upgrade

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.