Skip to content

Conversation

@petr-muller
Copy link
Member

@petr-muller petr-muller commented Jun 27, 2023

This PR is a part of the same effort like openshift/ci-tools#3512

OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.

All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version (which all Hive clusters are).

  1. Introduce a configure-cincinnati step that uses oc to set a cincinnati instance the cluster should use, and also to switch a channel to a desired one of the same version if a cluster is configured to follow a channel (otherwise we do not know what version to use).
  2. Make generic-claim workflow to use this step to switch Hive-provisioned clusters to use integration OSUS instance and
    a candidate channel.

This should not have any observable effect on jobs, it is just a mean to get more traffic to the integration OSUS instance.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 27, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 27, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@petr-muller
Copy link
Member Author

/pj-rehearse

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2023
@petr-muller petr-muller force-pushed the use-integration-cincinnati-in-hive-clusters branch from 96d5858 to 85925f8 Compare June 27, 2023 13:50
@petr-muller
Copy link
Member Author

/pj-rehearse

@petr-muller
Copy link
Member Author

/test all

@petr-muller
Copy link
Member Author

/pj-rehearse pull-ci-openshift-cincinnati-operator-master-operator-e2e-410

@openshift-ci openshift-ci bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2023
@petr-muller
Copy link
Member Author

/pj-rehearse pull-ci-redhat-developer-gitops-operator-master-v4.11-kuttl-sequential

@petr-muller petr-muller force-pushed the use-integration-cincinnati-in-hive-clusters branch from 92d9cd2 to 8a586c0 Compare June 27, 2023 16:38
@petr-muller
Copy link
Member Author

/test all
/pj-rehearse pull-ci-openshift-cincinnati-operator-master-operator-e2e-410 pull-ci-redhat-developer-gitops-operator-master-v4.11-kuttl-sequential

@petr-muller petr-muller force-pushed the use-integration-cincinnati-in-hive-clusters branch from 8a586c0 to 800795d Compare July 12, 2023 17:10
@petr-muller petr-muller changed the title o/cincinnati-operator: point the cluster at integration cincinnati generic-claim: use integration cincinnati instance in hive clusters Jul 12, 2023
@petr-muller petr-muller marked this pull request as ready for review July 12, 2023 17:18
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 12, 2023
@openshift-ci openshift-ci bot requested review from deepsm007 and neisw July 12, 2023 17:20
@petr-muller
Copy link
Member Author

/pj-rehearse pull-ci-openshift-cincinnati-master-e2e pull-ci-openshift-special-resource-operator-release-4.13-e2e-aws-ocp periodic-ci-openshift-knative-serverless-operator-main-4.13-upstream-e2e-kafka-aws-ocp-413-continuous pull-ci-openshift-knative-serverless-operator-main-4.13-ui-tests

@petr-muller
Copy link
Member Author

/retest

@petr-muller
Copy link
Member Author

/cc @openshift/test-platform @wking
/uncc @deepsm007 @neisw

DPTP for ci-operator/step-registry/generic-claim and Trevor for ci-operator/step-registry/openshift

@openshift-ci openshift-ci bot requested review from a team and wking and removed request for deepsm007 and neisw July 18, 2023 17:49
@petr-muller petr-muller changed the title generic-claim: use integration cincinnati instance in hive clusters OTA-905: generic-claim: use integration cincinnati instance in hive clusters Jul 18, 2023
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 18, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jul 18, 2023

@petr-muller: This pull request references OTA-905 which is a valid jira issue.

Details

In response to this:

OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.

All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version (which all Hive clusters are).

  1. Introduce a configure-cincinnati step that uses oc to set a cincinnati instance the cluster should use, and also to switch a channel to a desired one of the same version if a cluster is configured to follow a channel (otherwise we do not know what version to use).
  2. Make generic-claim workflow to use this step to switch Hive-provisioned clusters to use integration OSUS instance and
    a candidate channel.

This should not have any observable effect on jobs, it is just a mean to get more traffic to the integration OSUS instance.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jul 18, 2023

@petr-muller: This pull request references OTA-905 which is a valid jira issue.

Details

In response to this:

This PR is a part of the same effort like openshift/ci-tools#3512

OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.

All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version (which all Hive clusters are).

  1. Introduce a configure-cincinnati step that uses oc to set a cincinnati instance the cluster should use, and also to switch a channel to a desired one of the same version if a cluster is configured to follow a channel (otherwise we do not know what version to use).
  2. Make generic-claim workflow to use this step to switch Hive-provisioned clusters to use integration OSUS instance and
    a candidate channel.

This should not have any observable effect on jobs, it is just a mean to get more traffic to the integration OSUS instance.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@petr-muller petr-muller force-pushed the use-integration-cincinnati-in-hive-clusters branch from 800795d to a5e0b8d Compare July 18, 2023 17:55
OTA maintains staging (identical to production) and integration (running
on engineering candidate OCP clusters) Cincinnati instances. We are
searching for traffic that we could route to especially the integration
one, so that we can find possible problems with engineering candidate
early.

All the instances are serving identical data (up to some minimal skew
coming from when individual instance scrape their source data) so we
should be able to easily use the integration instance in CI clusters
that are running a released OCP version (which all Hive clusters are).

1. Introduce a `configure-cincinnati` step that uses `oc` to set a
   cincinnati instance the cluster should use, and also to switch
   a channel to a desired one of the same version if a cluster
   is configured to follow a channel (otherwise we do not know what
   version to use).
2. Make `generic-claim` workflow to use this step to switch
   Hive-provisioned clusters to use integration OSUS instance and
   a candidate channel.

This should not have any observable effect on jobs, it is just a mean to
get more traffic to the integration OSUS instance.
@petr-muller petr-muller force-pushed the use-integration-cincinnati-in-hive-clusters branch from a5e0b8d to 5ce92dc Compare July 19, 2023 17:52
@petr-muller
Copy link
Member Author

/pj-rehearse ack

@openshift-ci-robot openshift-ci-robot added rehearsals-ack Signifies that rehearsal jobs have been acknowledged and removed rehearsals-ack Signifies that rehearsal jobs have been acknowledged labels Jul 19, 2023
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@petr-muller: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-ViaQ-vector-v0.21-rh-cluster-logging-operator-e2e ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-v0.21-rh-clo-functional ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-release-5.6-cluster-logging-operator-e2e-5-6 ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-release-5.6-clo-functional-5-6 ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-release-5.7-cluster-logging-operator-e2e ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-release-5.7-clo-functional ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-release-5.8-cluster-logging-operator-e2e ViaQ/vector presubmit Registry content changed
pull-ci-ViaQ-vector-release-5.8-clo-functional ViaQ/vector presubmit Registry content changed
pull-ci-openshift-cincinnati-operator-master-operator-e2e-410 openshift/cincinnati-operator presubmit Registry content changed
pull-ci-openshift-cincinnati-operator-master-operator-e2e-latest-osus-410 openshift/cincinnati-operator presubmit Registry content changed
pull-ci-openshift-cincinnati-operator-master-operator-e2e-411 openshift/cincinnati-operator presubmit Registry content changed
pull-ci-openshift-cincinnati-operator-master-operator-e2e-latest-osus-411 openshift/cincinnati-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-master-e2e-ocp-target-minus-one openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-master-e2e-ocp-target-minus-two openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-master-functional openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-release-5.6-e2e-claim-aws openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-release-5.6-functional openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-release-5.7-e2e-ocp-target-minus-one openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-release-5.7-e2e-ocp-target-minus-two openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-release-5.7-functional openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-openshift-cluster-logging-operator-release-5.5-functional openshift/cluster-logging-operator presubmit Registry content changed
pull-ci-redhat-appstudio-managed-gitops-main-managed-gitops-e2e-tests redhat-appstudio/managed-gitops presubmit Registry content changed
pull-ci-stolostron-hypershift-addon-operator-main-e2e stolostron/hypershift-addon-operator presubmit Registry content changed
pull-ci-stolostron-hypershift-addon-operator-backplane-2.2-e2e stolostron/hypershift-addon-operator presubmit Registry content changed
pull-ci-stolostron-hypershift-addon-operator-backplane-2.4-e2e stolostron/hypershift-addon-operator presubmit Registry content changed
pull-ci-stolostron-hypershift-addon-operator-backplane-2.3-e2e stolostron/hypershift-addon-operator presubmit Registry content changed
pull-ci-stolostron-hypershift-addon-operator-backplane-2.1-e2e stolostron/hypershift-addon-operator presubmit Registry content changed
pull-ci-stolostron-hypershift-addon-operator-backplane-2.0-e2e stolostron/hypershift-addon-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-master-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-release-4.15-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-release-4.14-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-release-4.13-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-release-4.12-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-release-4.11-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed
pull-ci-openshift-special-resource-operator-release-4.10-e2e-aws-ocp openshift/special-resource-operator presubmit Registry content changed

A total of 483 jobs have been affected by this change. The above listing is non-exhaustive and limited to 35 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@petr-muller
Copy link
Member Author

/pj-rehearse ack

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jul 19, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 19, 2023

@petr-muller: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/redhat-developer/gitops-operator/master/v4.11-kuttl-sequential 8a586c087c002dd5fb4024f120a380fedf2b05f0 link unknown /pj-rehearse pull-ci-redhat-developer-gitops-operator-master-v4.11-kuttl-sequential
ci/rehearse/periodic-ci-openshift-knative-serverless-operator-main-4.13-upstream-e2e-kafka-aws-ocp-413-continuous 800795d3a0db147b72fed51b0b3410df998996d5 link unknown /pj-rehearse periodic-ci-openshift-knative-serverless-operator-main-4.13-upstream-e2e-kafka-aws-ocp-413-continuous
ci/rehearse/rh-ecosystem-edge/ci-tools-nvidia-gpu-operator/main/master-branch-gpu-e2e a5e0b8d2732ccd706734d3aaac7a383dbb60a475 link unknown /pj-rehearse pull-ci-rh-ecosystem-edge-ci-tools-nvidia-gpu-operator-main-master-branch-gpu-e2e
ci/rehearse/openshift-knative/client/release-v1.5/e2e-aws-ocp-410 a5e0b8d2732ccd706734d3aaac7a383dbb60a475 link unknown /pj-rehearse pull-ci-openshift-knative-client-release-v1.5-e2e-aws-ocp-410
ci/rehearse/openshift-knative/client/release-v1.6/e2e-aws-ocp-410 a5e0b8d2732ccd706734d3aaac7a383dbb60a475 link unknown /pj-rehearse pull-ci-openshift-knative-client-release-v1.6-e2e-aws-ocp-410
ci/rehearse/openshift-knative/client/release-v1.7/e2e-aws-ocp-410 a5e0b8d2732ccd706734d3aaac7a383dbb60a475 link unknown /pj-rehearse pull-ci-openshift-knative-client-release-v1.7-e2e-aws-ocp-410
ci/rehearse/openshift-knative/client/release-v1.4/e2e-aws-ocp-410 a5e0b8d2732ccd706734d3aaac7a383dbb60a475 link unknown /pj-rehearse pull-ci-openshift-knative-client-release-v1.4-e2e-aws-ocp-410
ci/rehearse/rh-ecosystem-edge/ci-tools-nvidia-gpu-operator/main/test-commit a5e0b8d2732ccd706734d3aaac7a383dbb60a475 link unknown /pj-rehearse pull-ci-rh-ecosystem-edge-ci-tools-nvidia-gpu-operator-main-test-commit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@smg247
Copy link
Member

smg247 commented Jul 19, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 19, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 19, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, smg247

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 19, 2023
@openshift-merge-robot openshift-merge-robot merged commit 536a839 into openshift:master Jul 19, 2023
petr-muller added a commit to petr-muller/release that referenced this pull request Jul 21, 2023
…sible

This PR is a part of the same effort as openshift#40703 and implements the same change as openshift#40711. This change is a separate step to avoid mixing up rehearsals.

OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.

All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these need to not query OSUS, otherwise
they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since openshift#8631

We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS.
petr-muller added a commit to petr-muller/release that referenced this pull request Aug 10, 2023
This PR is a part of the same effort as openshift#40703

OTA maintains staging (identical to production) and integration (running
on engineering candidate OCP clusters) Cincinnati instances. We are
searching for traffic that we could route to especially the integration
one, so that we can find possible problems with engineering candidate
early.

All the instances are serving identical data (up to some minimal skew
coming from when individual instance scrape their source data) so we
should be able to easily use the integration instance in CI clusters
that are running a released OCP version. Most CI clusters are not
running such versions, and these must *not* query OSUS, otherwise they
would trip an alert, causing noise in CI jobs. CI clusters are not
querying OSUS since openshift#8631

We can enhance the logic to validate whether the version we are going
to install is known to OSUS. If it is, we know we are installing a
published OCP version and it is safe to let the cluster query OSUS. We
still clear the channel to prevent the cluster from querying OSUS if we
the version we install is not known to OSUS, or if this job involves
upgrading the cluster to another version.
openshift-merge-robot pushed a commit that referenced this pull request Aug 10, 2023
#40711)

This PR is a part of the same effort as #40703

OTA maintains staging (identical to production) and integration (running
on engineering candidate OCP clusters) Cincinnati instances. We are
searching for traffic that we could route to especially the integration
one, so that we can find possible problems with engineering candidate
early.

All the instances are serving identical data (up to some minimal skew
coming from when individual instance scrape their source data) so we
should be able to easily use the integration instance in CI clusters
that are running a released OCP version. Most CI clusters are not
running such versions, and these must *not* query OSUS, otherwise they
would trip an alert, causing noise in CI jobs. CI clusters are not
querying OSUS since #8631

We can enhance the logic to validate whether the version we are going
to install is known to OSUS. If it is, we know we are installing a
published OCP version and it is safe to let the cluster query OSUS. We
still clear the channel to prevent the cluster from querying OSUS if we
the version we install is not known to OSUS, or if this job involves
upgrading the cluster to another version.
petr-muller added a commit to petr-muller/release that referenced this pull request Aug 14, 2023
…s, where possible

This PR is a part of the same effort as openshift#40703 and implements the same change as openshift#40711. This change is a separate step to avoid mixing up rehearsals.

OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.

All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these need to not query OSUS, otherwise they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since openshift#8631

We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS.
petr-muller added a commit to petr-muller/release that referenced this pull request Aug 17, 2023
…sible

This PR is a part of the same effort as openshift#40703 and implements the same change as openshift#40711. This change is a separate step to avoid mixing up rehearsals.

OTA maintains staging (identical to production) and integration (running on engineering candidate OCP clusters) Cincinnati instances. We are searching for traffic that we could route to especially the integration one, so that we can find possible problems with engineering candidate early.

All the instances are serving identical data (up to some minimal skew coming from when individual instance scrape their source data) so we should be able to easily use the integration instance in CI clusters that are running a released OCP version. Most CI clusters are not running such versions, and these need to not query OSUS, otherwise
they would trip an alert, causing noise in CI jobs. CI clusters are not querying OSUS since openshift#8631

We can enhance the logic to validate whether the version we are going to install is known to OSUS. If it is, we know we are installing a published OCP version and it is safe to let the cluster query OSUS. We still clear the channel to prevent the cluster from querying OSUS if we the version we install is not known to OSUS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants