Bug 2109374: pkg/clusterconditions/promql: Cap PromQL queries at 5 minutes #806
Conversation
In some clusters, these PromQL queries can hang for hours, possibly forever [1]. I think we have a 30s default KeepAlive timeout [2], but apparently there's enough socket traffic to keep from tripping that. This adds a 5m cap to the PromQL calls, although I'm not particularly attached to that particular number. We can always raise it if we start seeing timeouts in Insights for queries where taking that long seems reasonable. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2109374#c12 [2]: https://pkg.go.dev/github.com/prometheus/client_golang/api#pkg-variables
openshift-ci
bot
added
bugzilla/severity-high
|
@wking: This pull request references Bugzilla bug 2109374, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci
bot
added
the
approved
|
/retest |
1 similar comment
|
/retest |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jottofar, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hitting same old |
|
@wking: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@wking: All pull requests linked via external trackers have merged: Bugzilla bug 2109374 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Verified via regression testing. /cherrypick release-4.11 |
|
@wking: new pull request created: #815 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
5 participants
In some clusters, these PromQL queries can hang for hours, possibly forever. I think we have a 30s default
KeepAlivetimeout, but apparently there's enough socket traffic to keep from tripping that. This adds a 5m cap to the PromQL calls, although I'm not particularly attached to that particular number. We can always raise it if we start seeing timeouts in Insights for queries where taking that long seems reasonable.