OCPBUGS-32242: The MCO makes a lot of API requests for ClusterOperator objects #4321

djoshy · 2024-04-15T20:44:51Z

I added a lister and cached the CO object wherever I could, and only did a get call when there is an update conflict error. Initial results look promising. I think we should be safe to merge this if the test suite is passing, but it can wait till after FF. It does not introduce any new behavior, so we could probably skip pre-merge QE.

For a recentish test run(LARGE FILE WARNING) on #4327 :

{
                    "GroupVersionResource": {
                        "Group": "config.openshift.io",
                        "Version": "v1",
                        "Resource": "clusteroperators"
                    },
                    "RequestCounts": {
                        "SerializedRequestCounts": {
                            "RequestStartedCount": 0,
                            "RequestFinishedCount": 8034,
                            "ClientFailedRequestCount": 0,
                            "ServerFailedRequestCount": 0,
                            "PerHTTPStatusRequestCount": [
                                {
                                    "HTTPStatus": 200,
                                    "Count": 8042
                                }
                            ]
                        },
                        "PerVerbRequestCount": [
                            {
                                "Verb": "get",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 5933,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 5933
                                        }
                                    ]
                                }
                            },
                            {
                                "Verb": "update",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 2094,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 2094
                                        }
                                    ]
                                }
                            },
                            {
                                "Verb": "watch",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 7,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 15
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },

On a test run on this PR, the above scrape only shows:

{
                    "GroupVersionResource": {
                        "Group": "config.openshift.io",
                        "Version": "v1",
                        "Resource": "clusteroperators"
                    },
                    "RequestCounts": {
                        "SerializedRequestCounts": {
                            "RequestStartedCount": 0,
                            "RequestFinishedCount": 7,
                            "ClientFailedRequestCount": 0,
                            "ServerFailedRequestCount": 0,
                            "PerHTTPStatusRequestCount": [
                                {
                                    "HTTPStatus": 200,
                                    "Count": 15
                                }
                            ]
                        },
                        "PerVerbRequestCount": [
                            {
                                "Verb": "watch",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 7,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 15
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },

This is likely because a large amount of the calls have been optimized out; and some amount of calls when the cluster is born have been scrolled off in the audit logs and was not caught by the scraper tool. We should be failing most of the tests if we are not doing any update/get calls :P

openshift-ci-robot · 2024-04-15T20:45:01Z

@djoshy: This pull request references Jira Issue OCPBUGS-32242, which is invalid:

expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-04-15T20:45:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

djoshy · 2024-04-15T20:45:52Z

/test unit
/test verify
/test e2e-gcp-op

djoshy · 2024-04-15T21:01:20Z

/test unit
/test verify
/test e2e-gcp-op

djoshy · 2024-04-15T21:04:07Z

/test unit
/test verify
/test e2e-gcp-op

openshift-ci-robot · 2024-04-16T21:28:05Z

@djoshy: This pull request references Jira Issue OCPBUGS-32242. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-04-16T21:28:57Z

@djoshy: This pull request references Jira Issue OCPBUGS-32242, which is invalid:

expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2024-04-16T21:29:04Z

/test unit
/test verify
/test e2e-gcp-op

djoshy · 2024-04-16T21:32:31Z

/test unit
/test verify
/test e2e-gcp-op

djoshy · 2024-04-18T18:53:09Z

/test unit
/test verify
/test e2e-gcp-op

djoshy · 2024-04-18T19:17:50Z

/test e2e-aws-ovn

djoshy · 2024-04-19T11:16:23Z

/test unit
/test verify
/test e2e-gcp-op
/test e2e-aws-ovn

djoshy · 2024-04-19T15:03:47Z

/test e2e-aws-ovn

djoshy · 2024-04-19T20:21:13Z

/test unit
/test verify
/test e2e-gcp-op
/test e2e-aws-ovn

openshift-ci · 2024-05-15T16:29:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2024-05-17T09:45:24Z

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

openshift-ci-robot · 2024-05-17T09:45:56Z

@openshift-bot: This pull request references Jira Issue OCPBUGS-32242, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2024-05-17T15:04:31Z

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

openshift-ci-robot · 2024-05-17T15:05:53Z

@openshift-bot: This pull request references Jira Issue OCPBUGS-32242, which is invalid:

expected the bug to target either version "4.17." or "openshift-4.17.", but it targets "4.16.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2024-05-20T13:22:36Z

/jira refresh

openshift-ci-robot · 2024-05-20T13:22:41Z

@djoshy: This pull request references Jira Issue OCPBUGS-32242, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.17.0) matches configured target version for branch (4.17.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2024-05-20T13:23:48Z

/test all

sergiordlr · 2024-05-20T17:23:45Z

Critical e2e test cases executed in: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-ipi-f9-longduration-mco-critical/1792602290306158592 (failed, could not install)

sergiordlr · 2024-05-21T07:57:11Z

Critical e2e test cases executed in: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-ipi-f9-longduration-mco-critical/1792826455558721536 (installation failed)

sergiordlr · 2024-05-22T14:36:30Z

Critical test cases passed: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-ipi-f9-longduration-mco-critical/1792945702985273344

sergiordlr · 2024-05-22T14:54:02Z

Verfication steps

Critical test cases have passed
In the current CI job "e2e-aws-ovn" execution in we can find in the artifacts the audit-log-summary file https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_machine-config-operator/4321/pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn/1792546860762664960/artifacts/e2e-aws-ovn/openshift-e2e-test/artifacts/junit/audit-log-summary__20240520-141441.json

We can observe that for the clusteroperators

                {
                    "GroupVersionResource": {
                        "Group": "config.openshift.io",
                        "Version": "v1",
                        "Resource": "clusteroperators"
                    },
                    "RequestCounts": {
                        "SerializedRequestCounts": {
                            "RequestStartedCount": 0,
                            "RequestFinishedCount": 4,
                            "ClientFailedRequestCount": 0,
                            "ServerFailedRequestCount": 0,
                            "PerHTTPStatusRequestCount": [
                                {
                                    "HTTPStatus": 200,
                                    "Count": 9
                                }
                            ]
                        },
                        "PerVerbRequestCount": [
                            {
                                "Verb": "watch",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 4,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 9
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },

Those numbers are lower than the ones found in previous executions and reported at the beginning of this PR.

/label qe-approved

djoshy · 2024-05-22T14:57:01Z

/unhold

openshift-ci-robot · 2024-05-22T15:16:16Z

/retest-required

Remaining retests: 0 against base HEAD e50be3e and 2 for PR HEAD 461a468 in total

djoshy · 2024-05-22T17:41:55Z

/test all

djoshy · 2024-05-22T18:56:07Z

/test e2e-gcp-op-single-node

openshift-ci-robot · 2024-05-22T21:15:01Z

@djoshy: Jira Issue OCPBUGS-32242: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#4321

Jira Issue OCPBUGS-32242 has been moved to the MODIFIED state.

Details

In response to this:

I added a lister and cached the CO object wherever I could, and only did a get call when there is an update conflict error. Initial results look promising. I think we should be safe to merge this if the test suite is passing, but it can wait till after FF. It does not introduce any new behavior, so we could probably skip pre-merge QE.

For a recentish test run(LARGE FILE WARNING) on #4327 :

{
                   "GroupVersionResource": {
                       "Group": "config.openshift.io",
                       "Version": "v1",
                       "Resource": "clusteroperators"
                   },
                   "RequestCounts": {
                       "SerializedRequestCounts": {
                           "RequestStartedCount": 0,
                           "RequestFinishedCount": 8034,
                           "ClientFailedRequestCount": 0,
                           "ServerFailedRequestCount": 0,
                           "PerHTTPStatusRequestCount": [
                               {
                                   "HTTPStatus": 200,
                                   "Count": 8042
                               }
                           ]
                       },
                       "PerVerbRequestCount": [
                           {
                               "Verb": "get",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 5933,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 5933
                                       }
                                   ]
                               }
                           },
                           {
                               "Verb": "update",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 2094,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 2094
                                       }
                                   ]
                               }
                           },
                           {
                               "Verb": "watch",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 7,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 15
                                       }
                                   ]
                               }
                           }
                       ]
                   }
               },

On a test run on this PR, the above scrape only shows:

{
                   "GroupVersionResource": {
                       "Group": "config.openshift.io",
                       "Version": "v1",
                       "Resource": "clusteroperators"
                   },
                   "RequestCounts": {
                       "SerializedRequestCounts": {
                           "RequestStartedCount": 0,
                           "RequestFinishedCount": 7,
                           "ClientFailedRequestCount": 0,
                           "ServerFailedRequestCount": 0,
                           "PerHTTPStatusRequestCount": [
                               {
                                   "HTTPStatus": 200,
                                   "Count": 15
                               }
                           ]
                       },
                       "PerVerbRequestCount": [
                           {
                               "Verb": "watch",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 7,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 15
                                       }
                                   ]
                               }
                           }
                       ]
                   }
               },

This is likely because a large amount of the calls have been optimized out; and some amount of calls when the cluster is born have been scrolled off in the audit logs and was not caught by the scraper tool. We should be failing most of the tests if we are not doing any update/get calls :P

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-05-22T21:15:23Z

@djoshy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`6e194c2`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/okd-images	`461a468`	link	false	`/test okd-images`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-bot · 2024-05-23T10:32:30Z

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-machine-config-operator-container-v4.17.0-202405230209.p0.ge7da460.assembly.stream.el9 for distgit ose-machine-config-operator.
All builds following this will include this PR.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 15, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 15, 2024

djoshy force-pushed the reduce-optr-api-calls branch from 576ed4b to 1df281a Compare April 15, 2024 21:01

djoshy force-pushed the reduce-optr-api-calls branch from 1df281a to 6846332 Compare April 15, 2024 21:04

djoshy closed this Apr 16, 2024

djoshy force-pushed the reduce-optr-api-calls branch from 6846332 to 509a97d Compare April 16, 2024 21:28

djoshy reopened this Apr 16, 2024

djoshy force-pushed the reduce-optr-api-calls branch from 84dbb45 to 0cd0548 Compare April 16, 2024 21:31

djoshy force-pushed the reduce-optr-api-calls branch 2 times, most recently from e7db829 to eddfc35 Compare April 18, 2024 18:51

djoshy force-pushed the reduce-optr-api-calls branch from eddfc35 to 22bf993 Compare April 19, 2024 11:15

djoshy force-pushed the reduce-optr-api-calls branch from 22bf993 to cc03e88 Compare April 19, 2024 15:00

djoshy force-pushed the reduce-optr-api-calls branch from cc03e88 to 5060079 Compare April 19, 2024 19:53

openshift-ci bot assigned yuqi-zhang May 15, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 15, 2024

openshift-ci-robot added jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. and removed jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels May 17, 2024

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 20, 2024

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label May 22, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 22, 2024

openshift-merge-bot bot merged commit e7da460 into openshift:master May 22, 2024

djoshy deleted the reduce-optr-api-calls branch May 30, 2024 13:39

wking mentioned this pull request Jun 12, 2025

OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5119

Merged

OCPBUGS-32242: The MCO makes a lot of API requests for ClusterOperator objects #4321

OCPBUGS-32242: The MCO makes a lot of API requests for ClusterOperator objects #4321

Uh oh!

Conversation

djoshy commented Apr 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Apr 15, 2024

Uh oh!

openshift-ci bot commented Apr 15, 2024

Uh oh!

djoshy commented Apr 15, 2024

Uh oh!

djoshy commented Apr 15, 2024

Uh oh!

djoshy commented Apr 15, 2024

Uh oh!

openshift-ci-robot commented Apr 16, 2024

Uh oh!

openshift-ci-robot commented Apr 16, 2024

Uh oh!

djoshy commented Apr 16, 2024

Uh oh!

djoshy commented Apr 16, 2024

Uh oh!

djoshy commented Apr 18, 2024

Uh oh!

djoshy commented Apr 18, 2024

Uh oh!

djoshy commented Apr 19, 2024

Uh oh!

djoshy commented Apr 19, 2024

Uh oh!

djoshy commented Apr 19, 2024

Uh oh!

openshift-ci bot commented May 15, 2024

Uh oh!

openshift-bot commented May 17, 2024

Uh oh!

openshift-ci-robot commented May 17, 2024

Uh oh!

openshift-bot commented May 17, 2024

Uh oh!

openshift-ci-robot commented May 17, 2024

Uh oh!

djoshy commented May 20, 2024

Uh oh!

openshift-ci-robot commented May 20, 2024

Uh oh!

djoshy commented May 20, 2024

Uh oh!

sergiordlr commented May 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergiordlr commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergiordlr commented May 22, 2024

Uh oh!

sergiordlr commented May 22, 2024

Uh oh!

djoshy commented May 22, 2024

Uh oh!

openshift-ci-robot commented May 22, 2024

Uh oh!

djoshy commented May 22, 2024

Uh oh!

djoshy commented May 22, 2024

Uh oh!

openshift-ci-robot commented May 22, 2024

Uh oh!

openshift-ci bot commented May 22, 2024

Uh oh!

openshift-bot commented May 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

djoshy commented Apr 15, 2024 •

edited

Loading

sergiordlr commented May 20, 2024 •

edited

Loading

sergiordlr commented May 21, 2024 •

edited

Loading