Skip to content

Conversation

@anik120
Copy link
Contributor

@anik120 anik120 commented May 5, 2025

Polling with 30 secs timeout is causing thundering herd problem, causing the tests to timeout in resource constrained CI environments. This PR switches to using exponential backoffs with jitter.

@openshift-ci openshift-ci bot requested review from grokspawn and perdasilva May 5, 2025 19:36
@anik120 anik120 force-pushed the olmv1-tests-jitter branch from 4d8f791 to 60caeb5 Compare May 5, 2025 19:41
Factor: 1.5,
Jitter: 0.1,
Steps: 10,
Cap: 30 * time.Second,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be multiple minutes... at a minimum 2.

Polling with 30 secs timeout is causing thundering herd problem,
causing the tests to timeout in resource constrained CI environments.
This PR switches to using exponential backoffs with jitter.
@anik120 anik120 force-pushed the olmv1-tests-jitter branch from 60caeb5 to f8b59e8 Compare May 5, 2025 19:51
Copy link
Contributor

@tmshort tmshort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 5, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anik120, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 5, 2025
@anik120
Copy link
Contributor Author

anik120 commented May 5, 2025

retitle OCPBUGS-55712: (olmv1 tests) Use exponential backoffs for verifyAPIEndpoints

@anik120
Copy link
Contributor Author

anik120 commented May 5, 2025

/retitle OCPBUGS-55712: (olmv1 tests) Use exponential backoffs for verifyAPIEndpoints

@tmshort
Copy link
Contributor

tmshort commented May 5, 2025

/title OCPBUGS-55712: (olmv1 tests) Use exponential backoffs for verifyAPIEndpoints

@openshift-ci openshift-ci bot changed the title (olmv1 tests) Use exponential backoffs for verifyAPIEndpoints OCPBUGS-55712: (olmv1 tests) Use exponential backoffs for verifyAPIEndpoints May 5, 2025
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 5, 2025
@openshift-ci-robot
Copy link

@anik120: This pull request references Jira Issue OCPBUGS-55712, which is invalid:

  • expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Polling with 30 secs timeout is causing thundering herd problem, causing the tests to timeout in resource constrained CI environments. This PR switches to using exponential backoffs with jitter.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@grokspawn
Copy link
Contributor

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 5, 2025
@openshift-ci-robot
Copy link

@grokspawn: This pull request references Jira Issue OCPBUGS-55712, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.0) matches configured target version for branch (4.20.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD ce68c31 and 2 for PR HEAD f8b59e8 in total

1 similar comment
@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD ce68c31 and 2 for PR HEAD f8b59e8 in total

@tmshort
Copy link
Contributor

tmshort commented May 6, 2025

/retest-required

@tmshort
Copy link
Contributor

tmshort commented May 6, 2025

/retest-required

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD ce68c31 and 2 for PR HEAD f8b59e8 in total

2 similar comments
@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD ce68c31 and 2 for PR HEAD f8b59e8 in total

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD ce68c31 and 2 for PR HEAD f8b59e8 in total

@neisw
Copy link
Contributor

neisw commented May 8, 2025

/skip
/override ci/prow/e2e-metal-ipi-ovn-ipv6

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 8, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-metal-ipi-ovn-ipv6

Details

In response to this:

/skip
/override ci/prow/e2e-metal-ipi-ovn-ipv6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@neisw
Copy link
Contributor

neisw commented May 8, 2025

/override ci/prow/e2e-metal-ipi-ovn-ipv6

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 8, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-metal-ipi-ovn-ipv6

Details

In response to this:

/override ci/prow/e2e-metal-ipi-ovn-ipv6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD fa5164c and 2 for PR HEAD f8b59e8 in total

@neisw
Copy link
Contributor

neisw commented May 9, 2025

/override ci/prow/e2e-aws-ovn-edge-zones

known outage and has passed multiple times previously

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 9, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-aws-ovn-edge-zones

Details

In response to this:

/override ci/prow/e2e-aws-ovn-edge-zones

known outage and has passed multiple times previously

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@neisw
Copy link
Contributor

neisw commented May 9, 2025

/skip

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 73ca29c and 2 for PR HEAD f8b59e8 in total

@tmshort
Copy link
Contributor

tmshort commented May 9, 2025

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 9, 2025

@tmshort: tmshort unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers.

Details

In response to this:

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@neisw
Copy link
Contributor

neisw commented May 9, 2025

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 9, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-aws-ovn-edge-zones

Details

In response to this:

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 5c4033e and 2 for PR HEAD f8b59e8 in total

@neisw
Copy link
Contributor

neisw commented May 9, 2025

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 9, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-aws-ovn-edge-zones

Details

In response to this:

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tmshort
Copy link
Contributor

tmshort commented May 10, 2025

/retest-required

@tmshort
Copy link
Contributor

tmshort commented May 10, 2025

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 10, 2025

@tmshort: tmshort unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers.

Details

In response to this:

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 476ee38 and 1 for PR HEAD f8b59e8 in total

@neisw
Copy link
Contributor

neisw commented May 10, 2025

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 10, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-aws-ovn-edge-zones

Details

In response to this:

/override ci/prow/e2e-aws-ovn-edge-zones
/skip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tmshort
Copy link
Contributor

tmshort commented May 10, 2025

/retest-required

@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 10, 2025

@anik120: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-openstack-serial f8b59e8 link false /test e2e-openstack-serial
ci/prow/e2e-metal-ipi-serial f8b59e8 link false /test e2e-metal-ipi-serial
ci/prow/e2e-azure-ovn-upgrade f8b59e8 link false /test e2e-azure-ovn-upgrade
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview f8b59e8 link false /test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview
ci/prow/e2e-metal-ipi-ovn-dualstack f8b59e8 link false /test e2e-metal-ipi-ovn-dualstack
ci/prow/e2e-gcp-disruptive f8b59e8 link false /test e2e-gcp-disruptive
ci/prow/4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback f8b59e8 link false /test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
ci/prow/okd-scos-e2e-aws-ovn f8b59e8 link false /test okd-scos-e2e-aws-ovn
ci/prow/e2e-metal-ipi-serial-ovn-ipv6 f8b59e8 link false /test e2e-metal-ipi-serial-ovn-ipv6
ci/prow/e2e-aws-disruptive f8b59e8 link false /test e2e-aws-disruptive
ci/prow/e2e-gcp-fips-serial f8b59e8 link false /test e2e-gcp-fips-serial
ci/prow/e2e-gcp-ovn-etcd-scaling f8b59e8 link false /test e2e-gcp-ovn-etcd-scaling
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp-techpreview f8b59e8 link false /test e2e-metal-ipi-ovn-dualstack-bgp-techpreview
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6 f8b59e8 link false /test e2e-vsphere-ovn-dualstack-primaryv6
ci/prow/okd-e2e-gcp f8b59e8 link false /test okd-e2e-gcp
ci/prow/e2e-aws-ovn-kube-apiserver-rollout f8b59e8 link false /test e2e-aws-ovn-kube-apiserver-rollout
ci/prow/e2e-azure-ovn-etcd-scaling f8b59e8 link false /test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-metal-ipi-ovn-kube-apiserver-rollout f8b59e8 link false /test e2e-metal-ipi-ovn-kube-apiserver-rollout
ci/prow/e2e-metal-ipi-ovn-dualstack-local-gateway f8b59e8 link false /test e2e-metal-ipi-ovn-dualstack-local-gateway
ci/prow/e2e-metal-ipi-ovn f8b59e8 link false /test e2e-metal-ipi-ovn
ci/prow/e2e-metal-ipi-virtualmedia f8b59e8 link false /test e2e-metal-ipi-virtualmedia
ci/prow/e2e-vsphere-ovn-etcd-scaling f8b59e8 link false /test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-etcd-scaling f8b59e8 link false /test e2e-aws-ovn-etcd-scaling

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@tmshort
Copy link
Contributor

tmshort commented May 11, 2025

/retest-required

@openshift-merge-bot openshift-merge-bot bot merged commit 6724a84 into openshift:main May 11, 2025
56 checks passed
@openshift-ci-robot
Copy link

@anik120: Jira Issue OCPBUGS-55712: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-55712 has been moved to the MODIFIED state.

Details

In response to this:

Polling with 30 secs timeout is causing thundering herd problem, causing the tests to timeout in resource constrained CI environments. This PR switches to using exponential backoffs with jitter.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: openshift-enterprise-tests
This PR has been included in build openshift-enterprise-tests-container-v4.20.0-202505110811.p0.g6724a84.assembly.stream.el9.
All builds following this will include this PR.

@jianzhangbjz
Copy link
Member

The fixed PR for 4.19: #29770

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants