Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName #823

suleymanakbas91 · 2022-09-06T09:04:20Z

This PR adds two new alerting rules IngressWithoutClassName and UnmanagedRoutes that use the custom metrics created in
openshift/route-controller-manager#8.

IngressWithoutClassName fires when there is an Ingress with an unset IngressClassName for longer than one day.
UnmanagedRoutes fires when there is a Route owned by an Ingress no longer managed.

openshift-ci · 2022-09-06T09:04:24Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2022-09-06T09:04:27Z

@suleymanakbas91: This pull request references Bugzilla bug 1962502, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.12.0) matches configured target release for branch (4.12.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @quarterpin

Details

In response to this:

Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-09-06T12:16:32Z

@suleymanakbas91: This pull request references Bugzilla bug 1962502, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.12.0) matches configured target release for branch (4.12.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @quarterpin

Details

In response to this:

Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

suleymanakbas91 · 2022-09-06T15:20:10Z

/retest

candita · 2022-09-07T23:45:07Z

/assign
/assign @rfredette

suleymanakbas91 · 2022-09-12T13:28:23Z

/retest

suleymanakbas91 · 2022-09-13T09:18:22Z

/retest

brandisher · 2022-09-16T15:12:34Z

/retest

quarterpin · 2022-09-20T08:00:17Z

/label qe-approved

Verified via pre-merge workflow. For more details : https://bugzilla.redhat.com/show_bug.cgi?id=1962502#c7

brandisher · 2022-09-20T13:17:13Z

/retest

brandisher · 2022-09-20T16:58:42Z

/retest

brandisher · 2022-09-20T17:53:57Z

/retest

suleymanakbas91 · 2022-09-22T11:38:54Z

/retest

openshift-ci · 2022-10-05T14:42:07Z

@suleymanakbas91: This pull request references Bugzilla bug 1962502, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.12.0) matches configured target release for branch (4.12.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

Details

In response to this:

Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

candita · 2022-10-06T21:30:47Z

/retest-required

manifests/0000_90_ingress-operator_03_prometheusrules.yaml

candita · 2022-10-24T20:59:15Z

Strange test result. It doesn't like the Alerts, but I don't think this has anything to do with the new Alerts.

Oct 24 17:16:53.148: INFO: Alerts were detected during upgrade which are allowed:

alert etcdMemberCommunicationSlow pending for 120.93899989128113 seconds with labels: {To="762c0325ba8c065c", endpoint="etcd-metrics", instance="10.0.130.172:9979", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-130-172.us-west-1.compute.internal", service="etcd", severity="warning"} (allowed: Excluded because it triggers during upgrade (detects ~5m of high latency immediately preceeding the end of the test), and we don't want to change the alert because it is correct)
alert etcdMemberCommunicationSlow pending for 120.93899989128113 seconds with labels: {To="762c0325ba8c065c", endpoint="etcd-metrics", instance="10.0.190.160:9979", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-190-160.us-west-1.compute.internal", service="etcd", severity="warning"} (allowed: Excluded because it triggers during upgrade (detects ~5m of high latency immediately preceeding the end of the test), and we don't want to change the alert because it is correct)
Oct 24 17:16:53.149: FAIL: Unexpected alerts fired or pending during the upgrade:

alert TargetDown fired for 360 seconds with labels: {job="metrics", namespace="openshift-kube-scheduler-operator", service="metrics", severity="warning"}

/test e2e-aws-ovn-upgrade

suleymanakbas91 · 2022-10-25T09:38:36Z

I think you missed my point on this. Instead of testing openshift_ingress_to_route_controller_ingress_without_class_name{name=<...>}, try without the label, which is what the alert uses: openshift_ingress_to_route_controller_ingress_without_class_name. Let's find out if it aggregates in the case where there are multiples.

@candita The following image shows openshift_ingress_to_route_controller_ingress_without_class_name without any labels. It does not aggregate any one of them but just list them all seperately:

suleymanakbas91 · 2022-10-25T09:39:53Z

Failures seem to be unrelated..

/retest

suleymanakbas91 · 2022-10-25T12:54:55Z

Unrelated failures..

/retest

candita · 2022-10-25T15:27:06Z

/approve

candita · 2022-10-25T15:27:58Z

/test e2e-aws-operator

openshift-ci · 2022-10-25T15:28:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: candita

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [candita]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

brandisher · 2022-10-25T17:40:04Z

/lgtm

brandisher · 2022-10-25T18:31:40Z

/test e2e-aws-operator

suleymanakbas91 · 2022-10-26T04:18:26Z

/retest

suleymanakbas91 · 2022-10-26T08:10:29Z

/retest

suleymanakbas91 · 2022-10-26T11:42:19Z

/retest

suleymanakbas91 · 2022-10-26T15:28:31Z

/retest

suleymanakbas91 · 2022-10-26T19:26:34Z

/retest

suleymanakbas91 · 2022-10-26T20:16:29Z

/retest

Miciah · 2022-10-27T02:38:38Z

/test e2e-aws-operator
/test e2e-azure-operator
/test e2e-gcp-operator

suleymanakbas91 · 2022-10-27T07:05:59Z

/retest

suleymanakbas91 · 2022-10-27T09:51:56Z

/retest

suleymanakbas91 · 2022-10-27T12:55:07Z

/retest

Miciah · 2022-10-27T17:00:42Z

e2e-aws-operator failed because kube-apiserver reported NodeInstallerProgressing. Because (a) the e2e-azure-operator and e2e-gcp-operator jobs passed, (b) we know that e2e-aws-operator is flaky and that the failure we saw is a frequently occurring flake, and (c) the change in this PR is low risk, I'm going to override e2e-aws-operator.
/override ci/prow/e2e-aws-operator

openshift-ci · 2022-10-27T17:01:03Z

@Miciah: Overrode contexts on behalf of Miciah: ci/prow/e2e-aws-operator

Details

In response to this:

e2e-aws-operator failed because kube-apiserver reported NodeInstallerProgressing. Because (a) the e2e-azure-operator and e2e-gcp-operator jobs passed, (b) we know that e2e-aws-operator is flaky and that the failure we saw is a frequently occurring flake, and (c) the change in this PR is low risk, I'm going to override e2e-aws-operator.
/override ci/prow/e2e-aws-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-10-27T17:07:28Z

@suleymanakbas91: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci · 2022-10-27T17:10:59Z

@suleymanakbas91: All pull requests linked via external trackers have merged:

Bugzilla bug 1962502 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 6, 2022

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Sep 6, 2022

openshift-ci bot requested a review from quarterpin September 6, 2022 09:04

suleymanakbas91 force-pushed the route-ingress-metrics branch from 9916d88 to 1898399 Compare September 6, 2022 12:13

suleymanakbas91 marked this pull request as ready for review September 6, 2022 12:16

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 6, 2022

openshift-ci bot requested review from lmzuccarelli and miheer September 6, 2022 12:17

openshift-ci bot assigned candita and rfredette Sep 7, 2022

suleymanakbas91 force-pushed the route-ingress-metrics branch from 1898399 to 959b73b Compare September 12, 2022 18:48

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Sep 20, 2022

openshift-ci bot requested a review from zhaozhanqi October 5, 2022 14:47

candita reviewed Oct 6, 2022

View reviewed changes

manifests/0000_90_ingress-operator_03_prometheusrules.yaml Outdated Show resolved Hide resolved

openshift-ci bot mentioned this pull request Oct 25, 2022

Bug 1962502: Create custom metrics to list unmanaged Routes and Ingresses without IngressClassName openshift/route-controller-manager#8

Merged

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 25, 2022

openshift-ci bot assigned brandisher Oct 25, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 25, 2022

openshift-merge-robot merged commit 9c4689f into openshift:master Oct 27, 2022

suleymanakbas91 deleted the route-ingress-metrics branch October 27, 2022 17:12

Miciah mentioned this pull request Oct 27, 2022

[release-4.11] OCPBUGS-853: certificate-publisher: Don't publish extraneous certificates #824

Merged

Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName #823

Bug 1962502: Add alerts for unmanaged Routes and Ingresses without IngressClassName #823

Uh oh!

Conversation

suleymanakbas91 commented Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Sep 6, 2022

Uh oh!

openshift-ci bot commented Sep 6, 2022

Uh oh!

openshift-ci bot commented Sep 6, 2022

Uh oh!

suleymanakbas91 commented Sep 6, 2022

Uh oh!

candita commented Sep 7, 2022

Uh oh!

suleymanakbas91 commented Sep 12, 2022

Uh oh!

suleymanakbas91 commented Sep 13, 2022

Uh oh!

brandisher commented Sep 16, 2022

Uh oh!

quarterpin commented Sep 20, 2022

Uh oh!

brandisher commented Sep 20, 2022

Uh oh!

brandisher commented Sep 20, 2022

Uh oh!

brandisher commented Sep 20, 2022

Uh oh!

suleymanakbas91 commented Sep 22, 2022

Uh oh!

openshift-ci bot commented Oct 5, 2022

Uh oh!

candita commented Oct 6, 2022

Uh oh!

Uh oh!

candita commented Oct 24, 2022

Uh oh!

suleymanakbas91 commented Oct 25, 2022

Uh oh!

suleymanakbas91 commented Oct 25, 2022

Uh oh!

suleymanakbas91 commented Oct 25, 2022

Uh oh!

candita commented Oct 25, 2022

Uh oh!

candita commented Oct 25, 2022

Uh oh!

openshift-ci bot commented Oct 25, 2022

Uh oh!

brandisher commented Oct 25, 2022

Uh oh!

brandisher commented Oct 25, 2022

Uh oh!

suleymanakbas91 commented Oct 26, 2022

Uh oh!

suleymanakbas91 commented Oct 26, 2022

Uh oh!

suleymanakbas91 commented Oct 26, 2022

Uh oh!

suleymanakbas91 commented Oct 26, 2022

Uh oh!

suleymanakbas91 commented Oct 26, 2022

Uh oh!

suleymanakbas91 commented Oct 26, 2022

Uh oh!

Miciah commented Oct 27, 2022

Uh oh!

suleymanakbas91 commented Oct 27, 2022

Uh oh!

suleymanakbas91 commented Oct 27, 2022

Uh oh!

suleymanakbas91 commented Oct 27, 2022

Uh oh!

Miciah commented Oct 27, 2022

Uh oh!

openshift-ci bot commented Oct 27, 2022

Uh oh!

suleymanakbas91 commented Sep 6, 2022 •

edited

Loading