allow loki target to be down#26771
Conversation
openshift-ci
bot
added
the
approved
test/e2e/upgrade/alert/alert.go
Outdated
| allowedFiringAlerts := helper.MetricConditions{ | ||
| { | ||
| Selector: map[string]string{"alertname": "TargetDown", "namespace": "openshift-e2e-loki"}, | ||
| Text: "Loki is nice to have, but we can allow it to be down briefly", |
There was a problem hiding this comment.
TargetDown is already for: 15m, so not clear to me that we can claim "briefly" when allowing it to fire. Dropping "briefly" and just saying ...but we can allow it to be down. would be internally consistent.
deads2k
force-pushed
the
ignore-promtaildown
branch
from
1f2c3fa to
e7e4c05
Compare
wking
left a comment
wking
left a comment
There was a problem hiding this comment.
Sounds like pending alerts are non-fatal today, so I don't think we need the allowedPendingAlerts exception, but doesn't matter too much to me, so:
/lgtm
/hold
Pull the hold if you want to keep the allowedPendingAlerts around in the window before we get a real fix and revert this, but looking at the run, there are a whole bunch of TargetDown pending alerts today during various phases of the updates, and we don't have explicit exceptions for those (because we're only concerned about alerts that are still pending when the update completes).
openshift-ci
bot
added
the
do-not-merge/hold
deads2k
force-pushed
the
ignore-promtaildown
branch
from
e7e4c05 to
9480da6
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
deads2k
added
bugzilla/valid-bug
|
this built. we want to make the next payload. merging. |
|
@deads2k: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
2 participants
prevent failures like https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-aws-ovn-upgrade/1484205905842016256