[release-4.13] OCPBUGS-14367: High API requests due to allowlist and operconfig reconcilers running too often #1824
Conversation
The allowlist reconciler was being triggered every time any configmap in any namespace changes, which effectively means the reconciler was repeatedly running thousands of times per hour. In addition to the CPU hit, the first thing done by the reconciler is a configmap GET request for the allowlist configmap, regardless of the triggering configmap, creating the allowlist configmap if it doesn't exist. This results in thousands of avoidable configmap API requests per hour. This commit updates the allowlist watcher to to use a configmap informer for a specific namespace, rather than triggering the reconciler whenever any configmap in any namespace changes. In addition, custom predicates filter requests for configmaps other than the allowlist or default allowlist. As a result, the allowlist reconciler is now only called if the allowlist configmap changes. Signed-off-by: Don Penney <dpenney@redhat.com>
An earlier update added additional watchers to the operconfig reconciler, watching for configmap and node changes, with a request transformer that sets the network operator namespace along with network name. However, because the network object isn't namespaced, the reconciler is effectively called with two different request objects, returning a RequeueAfter on every call. This results in two separate recurring reconcilers running every 3 minutes, rather than one. This update modifies the reconcileOperConfig request transformer to leave the namespace field unset, matching the network object reconcile request. This ensures there is only one recurring reconciler running. Signed-off-by: Don Penney <dpenney@redhat.com>
Removed the now-redundant namespace check from the allowlist controller reconciler, as it is now triggered by a configmap informer that is watching only the specific namespace that was being checked, per review request. Additionally, changed a log from previous to klog to be consistent with allowlist controller, also per review request. Signed-off-by: Don Penney <dpenney@redhat.com>
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-11565 has been cloned as Jira Issue OCPBUGS-14367. Will retitle bug to link to clone. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/severity-moderate
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-14367, which is valid. The bug has been moved to the POST state. 6 validation(s) were run on this bug
Requesting review from QA contact: The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retest |
|
/cc @mlguerrero12 |
|
/retest |
|
/lgtm |
|
/retest |
|
/approve |
|
/label backport-risk-assessed |
openshift-ci
bot
added
the
backport-risk-assessed
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dougbtv, mlguerrero12, openshift-cherrypick-robot The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@openshift-cherrypick-robot: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/label cherry-pick-approved |
openshift-ci
bot
added
the
cherry-pick-approved
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-14367: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-14367 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Fix included in accepted release 4.13.0-0.nightly-2023-06-02-132339 |
14 participants
This is an automated cherry-pick of #1788
/assign donpenney