OCPBUGS-49763: Disregard health endpoints in the burn rate alerts #1742
Conversation
openshift-ci
bot
added
the
approved
|
/cc |
|
/retitle OCPBUGS-42083: Disregard health endpoints in the burn rate alerts |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/severity-important
|
@dgrisonnet: This pull request references Jira Issue OCPBUGS-42083, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-bug
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
bot
added
the
lifecycle/stale
|
/retest |
|
/remove-lifecycle stale |
openshift-merge-robot
added
the
needs-rebase
openshift-ci
bot
removed
the
lifecycle/stale
Signed-off-by: Damien Grisonnet <[email protected]>
dgrisonnet
force-pushed
the
slo-health-endpoints
branch
from
c478ada to
c75a1dd
Compare
openshift-merge-robot
removed
the
needs-rebase
dgrisonnet
force-pushed
the
slo-health-endpoints
branch
from
c75a1dd to
0eb743e
Compare
dgrisonnet
changed the title
openshift-ci-robot
removed
the
jira/severity-important
|
@dgrisonnet: This pull request references Jira Issue OCPBUGS-49763, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@dgrisonnet: This pull request references Jira Issue OCPBUGS-49763, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retest-required |
|
So there's already some precedence for excluding endpoints: #1126. Is it known what causes the timeout of the removed endpoints? New revisions rolling out? Or, is there a different factor? Edit: https://issues.redhat.com/browse/OCPBUGS-42083?focusedId=25653649&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-25653649 mentions the rollouts. The comment also mentions this can only happen during bootstrap. Is there a chance the situation improved and this PR is mostly for future safety? |
|
The cause was etcd readiness checks failing during bootstrap. It is what started the discussion around removing the etcd rediness checks in kas |
|
I added some explanations on https://issues.redhat.com//browse/OCPBUGS-49763 |
benluddy
left a comment
benluddy
left a comment
There was a problem hiding this comment.
This change makes sense to me. Both /livez and /readyz respond with HTTP 500 as part of their normal operation. I don't think this creates a blind spot because latency and error codes in /healthz, /livez, and /readyz are positively correlated with latency and error codes in other endpoints. The same considerations don't apply to /statusz or /flagz.
/lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: benluddy, dgrisonnet The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/label acknowledge-critical-fixes-only |
openshift-ci
bot
added
the
acknowledge-critical-fixes-only
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
|
/unhold |
openshift-ci
bot
removed
the
do-not-merge/hold
|
/cherry-pick release 4.18 |
|
@dgrisonnet: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@dgrisonnet: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/retest-required |
openshift-merge-bot
bot
merged commit 730ff1e
into
openshift:master
|
@dgrisonnet: Jira Issue OCPBUGS-49763: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-49763 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@dgrisonnet: cannot checkout DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cherry-pick release-4.18 |
|
@dgrisonnet: #1742 failed to apply on top of branch "release-4.18": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
this also fixes a bad `le="30(.0)?"` as a result of openshift#1742 The chosen approach is to move to float buckets one Prometheus v3 is merged forgetting about the integer historical data. That will be done in openshift#1816
this also fixes a bad `le="30(.0)?"` as a result of openshift#1742 The chosen approach is to move to float buckets one Prometheus v3 is merged forgetting about the integer historical data. That will be done in openshift#1816
this also fixes a bad `le="30(.0)?"` as a result of openshift#1742 The chosen approach is to move to float buckets one Prometheus v3 is merged forgetting about the integer historical data. That will be done in openshift#1816
10 participants
While investigating https://issues.redhat.com//browse/OCPBUGS-42083, we noticed that some of the disruption was caused by 500 errors on health endpoints that were due to timeout, but the timeout for these endpoints is lower than it is for the other kube-apiserver endpoints.
Since we can already monitor these endpoints from the probes, I'd be better to remove them from the SLO alert to reduce its scope.