OTA-905: ipi install: use integration cincinnati in CI clusters, where possible #40711
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator |
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
b37c898 to
3056b83
Compare
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator |
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
3056b83 to
e8ef6f9
Compare
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator |
|
/test all |
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
e8ef6f9 to
77f0c0a
Compare
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator |
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
77f0c0a to
b6719a0
Compare
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator |
1 similar comment
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator |
|
This is what I wanted to see (from https://console-openshift-console.apps.build05.l9oh.p1.openshiftapps.com/k8s/ns/ci-op-zvddhzch/pods/maistra-envoy-unit-2-4-ipi-install-install/logs) |
There was a problem hiding this comment.
I dunno why folks have copy/pasted this around so much, but:
$ git --no-pager grep cvo-overrides.yaml ci-operator/step-registry/
ci-operator/step-registry/ipi/install/install/aws/ipi-install-install-aws-commands.sh:sed -i '/^ channel:/d' "${dir}/manifests/cvo-overrides.yaml"
ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh:sed -i '/^ channel:/d' "${dir}/manifests/cvo-overrides.yaml"
ci-operator/step-registry/ipi/install/libvirt/install/ipi-install-libvirt-install-commands.sh:sed -i '/^ channel:/d' ${dir}/manifests/cvo-overrides.yaml
ci-operator/step-registry/ipi/install/powervs/install/ipi-install-powervs-install-commands.sh:sed -i '/^ channel:/d' "${dir}/manifests/cvo-overrides.yaml"
ci-operator/step-registry/upi/conf/vsphere/platform-external/upi-conf-vsphere-platform-external-commands.sh:sed -i '/^ channel:/d' "manifests/cvo-overrides.yaml"
ci-operator/step-registry/upi/conf/vsphere/upi-conf-vsphere-commands.sh:sed -i '/^ channel:/d' "manifests/cvo-overrides.yaml"
ci-operator/step-registry/upi/conf/vsphere/zones/upi-conf-vsphere-zones-commands.sh:sed -i '/^ channel:/d' "manifests/cvo-overrides.yaml"
ci-operator/step-registry/upi/install/aws/cluster/upi-install-aws-cluster-commands.sh:sed -i '/^ channel:/d' ${ARTIFACT_DIR}/installer/manifests/cvo-overrides.yaml
ci-operator/step-registry/upi/install/azure/upi-install-azure-commands.sh:sed -i '/^ channel:/d' manifests/cvo-overrides.yamlI'm completely fine leaving those other steps alone, and letting their maintainers continue to copy/paste or find some new way to DRY things up, but wanted to point out the issue in case other folks wanted to do something more proactive.
There was a problem hiding this comment.
I think I will try to update them all, but I will do that in separate PRs otherwise the rehearsals will be pretty hard to track.
ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh
Outdated
Show resolved
Hide resolved
|
/test build04-dry |
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
b6719a0 to
5b5ca77
Compare
petr-muller
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
I cleaned up the change, added proper commit / pr description and I want to see some rehearsals. /hold |
openshift-ci
bot
added
the
do-not-merge/hold
I fear nothing! /pj-rehearse |
|
@petr-muller, If the problem persists, please contact Test Platform. |
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
b12d197 to
232f7dc
Compare
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 periodic-ci-dora-metrics-pelorus-master-4.13-e2e-openshift-test-scenario-1-periodic pull-ci-opendatahub-io-kubeflow-master-odh-notebook-controller-e2e pull-ci-3scale-3scale-operator-master-test-e2e pull-ci-openshift-coredns-master-e2e-aws-ovn pull-ci-medik8s-machine-deletion-remediation-main-4.14-openshift-e2e periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-upgrade periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ibmcloud-ovn-multi-s390x periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-sdn-arm64 |
1 similar comment
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 periodic-ci-dora-metrics-pelorus-master-4.13-e2e-openshift-test-scenario-1-periodic pull-ci-opendatahub-io-kubeflow-master-odh-notebook-controller-e2e pull-ci-3scale-3scale-operator-master-test-e2e pull-ci-openshift-coredns-master-e2e-aws-ovn pull-ci-medik8s-machine-deletion-remediation-main-4.14-openshift-e2e periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-upgrade periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ibmcloud-ovn-multi-s390x periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-sdn-arm64 |
This PR is a part of the same effort as openshift#40703 OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early. All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these must *not* query OSUS, otherwise they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since openshift#8631 We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS, or if this job involves upgrading the cluster to another version.
petr-muller
force-pushed
the
use-integration-cincinnati-in-more-ci-clusters
branch
from
232f7dc to
157b4b1
Compare
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
|
/pj-rehearse pull-ci-maistra-envoy-maistra-2.4-maistra-envoy-unit-2-4 periodic-ci-dora-metrics-pelorus-master-4.13-e2e-openshift-test-scenario-1-periodic pull-ci-opendatahub-io-kubeflow-master-odh-notebook-controller-e2e pull-ci-3scale-3scale-operator-master-test-e2e pull-ci-openshift-coredns-master-e2e-aws-ovn pull-ci-medik8s-machine-deletion-remediation-main-4.14-openshift-e2e periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-upgrade periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ibmcloud-ovn-multi-s390x periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-aws-sdn-arm64 |
|
[REHEARSALNOTIFIER]
A total of 12772 jobs have been affected by this change. The above listing is non-exhaustive and limited to 35 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@petr-muller: This pull request references OTA-905 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@petr-muller: This pull request references OTA-905 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgoodwin, petr-muller, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@petr-muller: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/pj-rehearse periodic-ci-openshift-multiarch-master-nightly-4.14-ocp-e2e-ibmcloud-ovn-multi-s390x |
|
/pj-rehearse ack |
openshift-ci-robot
added
the
rehearsals-ack
…s, where possible This PR is a part of the same effort as openshift#40703 and implements the same change as openshift#40711. This change is a separate step to avoid mixing up rehearsals. OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early. All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these need to not query OSUS, otherwise they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since openshift#8631 We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS.
…sible This PR is a part of the same effort as openshift#40703 and implements the same change as openshift#40711. This change is a separate step to avoid mixing up rehearsals. OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early. All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these need to not query OSUS, otherwise they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since openshift#8631 We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS.
5 participants
This PR is a part of the same effort as #40703
OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.
All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these need to not query OSUS, otherwise
they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since #8631
We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS.
Changing
ipi-install-install, this has a huge blast radius with these three potential failure cases:ipi-install-install-stableinitialsteps. I plan to monitor search.ci for an uptick of jobs whereCannotRetrieveUpdatesalert fires