Skip to content

Conversation

@xueqzhan
Copy link
Contributor

@xueqzhan xueqzhan commented Oct 20, 2022

Reverts #6124

Per OpenShift policy, we are reverting this breaking change to get CI and/or nightly payloads flowing again.

This has been affecting almost every CI and nightly payload since last night:

  1. 4.12.0-0.ci-2022-10-20-074427
  2. 4.12.0-0.nightly-2022-10-20-015716
  3. 4.12.0-0.nightly-2022-10-19-205630

The error message in all the cases are similar and in the following format:

level=info msg=Waiting up to 5m0s (until 10:12PM) to ensure each cluster operator has finished progressing...
level=error msg=Cluster operator node-tuning does not meet stability threshold of Progressing=false for greater than 30 seconds with Reason: "ProfileProgressing" and Message: "Waiting for 1/6 Profiles to be applied"

But it is affecting different operators and different jobs:

To unrevert this, revert this PR, and layer an additional separate commit on top that addresses the problem. Given this is affecting many payload jobs, but with different operators, we recommend running the following command on the unrevert at least two times to prove the fix:

/payload 4.12 nightly blocking

@sdodson
Copy link
Member

sdodson commented Oct 20, 2022

/lgtm
/approve
We'll bring this back in as soon as we've tuned it to a timeout which locks success at current CI levels as well as David's watch replay suggestions.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 20, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 20, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 20, 2022
@xueqzhan
Copy link
Contributor Author

/override ci/prow/e2e-gcp-ovn

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 20, 2022

@xueqzhan: Overrode contexts on behalf of xueqzhan: ci/prow/e2e-gcp-ovn

Details

In response to this:

/override ci/prow/e2e-gcp-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xueqzhan
Copy link
Contributor Author

/override ci/prow/e2e-vsphere-ovn

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 20, 2022

@xueqzhan: Overrode contexts on behalf of xueqzhan: ci/prow/e2e-vsphere-ovn

Details

In response to this:

/override ci/prow/e2e-vsphere-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dgoodwin
Copy link
Contributor

/override ci/prow/e2e-ibmcloud-ovn
/override ci/prow/e2e-libvirt
/override ci/prow/okd-e2e-aws-upgrade
/override ci/prow/okd-scos-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-upgrade
/override ci/prow/e2e-agent-compact
/override ci/prow/e2e-aws-ovn
/override ci/prow/e2e-azure-ovn
/override ci/prow/images
/override ci/prow/okd-images
/override ci/prow/okd-scos-images

Need to get this revert in asap, we're blocked without, and it's a clean revert back to known good state.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 20, 2022

@dgoodwin: Overrode contexts on behalf of dgoodwin: ci/prow/e2e-agent-compact, ci/prow/e2e-aws-ovn, ci/prow/e2e-azure-ovn, ci/prow/e2e-ibmcloud-ovn, ci/prow/e2e-libvirt, ci/prow/images, ci/prow/okd-e2e-aws-upgrade, ci/prow/okd-images, ci/prow/okd-scos-e2e-aws-ovn, ci/prow/okd-scos-e2e-aws-upgrade, ci/prow/okd-scos-images

Details

In response to this:

/override ci/prow/e2e-ibmcloud-ovn
/override ci/prow/e2e-libvirt
/override ci/prow/okd-e2e-aws-upgrade
/override ci/prow/okd-scos-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-upgrade
/override ci/prow/e2e-agent-compact
/override ci/prow/e2e-aws-ovn
/override ci/prow/e2e-azure-ovn
/override ci/prow/images
/override ci/prow/okd-images
/override ci/prow/okd-scos-images

Need to get this revert in asap, we're blocked without, and it's a clean revert back to known good state.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants