Skip to content

Bug 1932618: Don't allow alerts to fire during a test run #25923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
openshift-merge-robot merged 1 commit into from
Feb 25, 2021
Merged

Bug 1932618: Don't allow alerts to fire during a test run #25923

openshift-merge-robot merged 1 commit into from
Feb 25, 2021

Conversation

@smarterclayton
Copy link
Contributor

@smarterclayton smarterclayton commented Feb 24, 2021 •
edited
Loading

The alert test has not been working and a PR was able to merge that was firing alerts during the test run. https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-baremetal-operator/110/pull-ci-openshift-cluster-baremetal-operator-master-e2e-agnostic/1364398641615212544 shows the two firing alerts being bypassed by the query in the shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured late test

Fix the test logic based on:

  1. Remove the filter on KubeAPILatencyHigh - no longer needed in 4.6+
  2. Remove the filter on KubePodCrashLooping - no longer needed in 4.6+
  3. Sort the metric by number of seconds the alert was firing

The bug that allowed the merge was the filter for KubePodCrashLooping,
to omit a series we must use unless X. If we reintroduce a filter we
should use that syntax.

@smarterclayton
test: Don't allow alerts to fire during a test run
56609ae 
The alert test has not been working and a PR was able to merge that
was firing alerts during the test run.

Fix the test logic based on:

1. Remove the filter on KubeAPILatencyHigh - no longer needed in 4.6+
2. Remove the filter on KubePodCrashLooping - no longer needed in 4.6+
3. Sort the metric by number of seconds the alert was firing

The bug that allowed the merge was the filter for KubePodCrashLooping,
to omit a series we must use `unless X`. If we reintroduce a filter we
should use that syntax.
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Feb 24, 2021
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 24, 2021

@smarterclayton: This pull request references Bugzilla bug 1932618, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.8.0) matches configured target release for branch (4.8.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
Details

In response to this:

Bug 1932618: Don't allow alerts to fire during a test run

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Feb 24, 2021
wking
wking approved these changes Feb 24, 2021
View reviewed changes
Copy link
Member

@wking wking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 24, 2021
ravisantoshgudimetla
ravisantoshgudimetla approved these changes Feb 24, 2021
View reviewed changes
Copy link
Contributor

@ravisantoshgudimetla ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@wking wking mentioned this pull request Feb 24, 2021
Closed
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 24, 2021

@smarterclayton: This pull request references Bugzilla bug 1932618, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.8.0) matches configured target release for branch (4.8.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
Details

In response to this:

Bug 1932618: Don't allow alerts to fire during a test run

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 24, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ravisantoshgudimetla, smarterclayton, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

1 similar comment
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 24, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ravisantoshgudimetla, smarterclayton, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 24, 2021
@smarterclayton
Copy link
Contributor Author

smarterclayton commented Feb 24, 2021

/test e2e-gcp

@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 24, 2021

/retest

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 24, 2021

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 25, 2021

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 25, 2021

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 25, 2021 •
edited
Loading

@smarterclayton: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-disruptive 56609ae link /test e2e-aws-disruptive

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 25, 2021

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 25, 2021

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 7f0714b into openshift:master Feb 25, 2021
@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 25, 2021

@smarterclayton: All pull requests linked via external trackers have merged:

Bugzilla bug 1932618 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1932618: Don't allow alerts to fire during a test run

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lilic lilic Awaiting requested review from lilic

@s-urbaniak s-urbaniak Awaiting requested review from s-urbaniak

2 more reviewers

@wking wking wking approved these changes

@ravisantoshgudimetla ravisantoshgudimetla ravisantoshgudimetla approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@wking wking

@ravisantoshgudimetla ravisantoshgudimetla

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@smarterclayton @openshift-ci-robot @openshift-bot @wking @ravisantoshgudimetla @openshift-merge-robot