OTA-656: Add CI jobs for heterogeneous cluster upgrades #33761

petr-muller · 2022-11-04T13:24:20Z

multiarch: add scaffolding for hetero->hetero upgrade jobs
make update

openshift-ci-robot · 2022-11-04T13:25:10Z

@petr-muller: the following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-heterogeneous-upgrade	N/A	periodic	Periodic changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals

petr-muller · 2022-11-04T13:25:43Z

/pj-rehearse

petr-muller · 2022-11-04T13:46:00Z

   * failed to generate steps from config: failed to resolve release arm64-latest: failed to request latest release: server responded with 404: no release configuration exists with the requested name

It seems we do not have 4.13 heterogeneous releses yet...

petr-muller · 2022-11-04T14:07:30Z

/pj-rehearse

petr-muller · 2022-11-04T14:16:37Z

/pj-rehearse

petr-muller · 2022-11-04T15:57:43Z

Seeing some noise in the https://steps.ci.openshift.org/reference/ipi-install-heterogeneous step:

Creating aarch64 worker MachineSet
Warning: incorrect GroupVersionKind for AWSMachineProviderConfig object: machine.openshift.io/v1beta1, Kind=AWSMachineProviderConfig
machineset.machine.openshift.io/ci-op-8lix2l21-008f2-qtw4c-worker-us-west-2a-additional created
W1104 15:12:13.138293      96 loader.go:223] Config not found: /tmp/installer/auth/kubeconfig
Error from server (Forbidden): routes.route.openshift.io "console" is forbidden: User "system:serviceaccount:ci-op-8lix2l21:ocp-e2e-aws-ovn-heterogeneous-upgrade" cannot get resource "routes" in API group "route.openshift.io" in the namespace "openshift-console"
Exiting with ret=0

petr-muller · 2022-11-04T18:12:41Z

/pj-rehearse

petr-muller · 2022-11-07T16:07:52Z

nightly-4.11-ocp-e2e-aws-heterogeneous
a. templateinstance readiness test should report ready soon after all annotated objects are ready does not seem to be related but I want more runs.
b. Heterogeneous setup looks good
nightly-4.12-upgrade-from-nightly-4.11 looks good
nightly-4.12-upgrade-from-stable-4.11 went well but failed on CI infra flake in post phase
nightly-4.12-ocp-e2e-aws-ovn-heterogeneous
a. alert TelemeterClientFailures fired for 144 seconds with labels: {namespace="openshift-monitoring", severity="warning"} does not seem to be related but I want more runs
b. Warning: incorrect GroupVersionKind for AWSMachineProviderConfig object: machine.openshift.io/v1beta1, Kind=AWSMachineProviderConfig in a heterogeneous setup step seems to be in all initial 4.12 jobs, and does not seem to be caused by the step itself - this content is pulled from the cluster, modified and re-applied
nightly-4.13-* jobs all fail with failed to resolve release arm64-latest: failed to request latest release: server responded with 404: no release configuration exists with the requested name : there are no 4.13 releases on https://multi.ocp.releases.ci.openshift.org/ yet

petr-muller · 2022-11-07T16:16:12Z

ci-operator/step-registry/openshift/upgrade/aws/heterogeneous/OWNERS

I am not sure about a good set of reviewers here, this is a copy paste of others' upgrade workflow OWNERS file.

It is probably worth including someone from multi-arch in here, so they can approve new upgrade-aws-heterogeneous-* steps/chains/workflows if we need more in the future. @deepsm007 , would that be you? Or can you recommend someone else?

I don't think we need to block on having someone from multi-arch though. If we stick with the parent-dir approver set, hack/dedup-owners.sh makes it easy to set up a DRYer symlink.

@deepsm007 and @Prashanth684 to own heterogeneous

petr-muller · 2022-11-07T16:21:44Z

/pj-rehearse

petr-muller · 2022-11-07T16:37:34Z

/pj-rehearse

petr-muller · 2022-11-07T16:37:37Z

/uncc @fabianofranz @jaypoulz
/cc @deepsm007 @aleskandro @jottofar @wking

I think is is now reviewable. Several questions:

I selected random hours of the day for the cron patterns and set the day to Monday to avoid collisions with MIXEDARCH-37: Arm worker nodes for heterogeneous workflow #31965, but I do not have the full context about why the crons are actually needed, so I'd welcome insights about whether these are set appropriately
Is the scope correct? Do we need these jobs? Fewer of them? Are some missing?
Who are good owners for the workflow? OTA? Multiarch folks?

petr-muller · 2022-11-09T16:04:01Z

/pj-rehearse ack

wking

We can refresh the LGTM after the OWNERS tweak.

/approve
/hold for OWNERS tweak

- `$KUBECONFIG` is set by ci-operator - Console URL extraction only needs to be done by steps that *install* a cluster

openshift-ci · 2022-11-10T13:40:16Z

@petr-muller: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.12-upgrade-from-nightly-4.11-ocp-ovn-remote-libvirt-ppc64le	f4c63fc640a1d62e9ca04fa720ff240df86c9bc5	link	unknown	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.12-upgrade-from-nightly-4.11-ocp-e2e-aws-heterogeneous-upgrade	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-stable-4.12-ocp-e2e-aws-heterogeneous-upgrade	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.12-ocp-e2e-aws-ovn-heterogeneous-upgrade	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-nightly-4.12-ocp-ovn-remote-libvirt-s390x	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-nightly-4.12-ocp-e2e-aws-sdn-arm64	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-heterogeneous-upgrade	b4dc07d5dc48e31b07598bca18ba10d6c0d4d24d	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-nightly-4.12-ocp-e2e-aws-heterogeneous-upgrade	b4dc07d5dc48e31b07598bca18ba10d6c0d4d24d	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.12-multi-nightly-e2e-aws-ipi-p2	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-upgrade-from-nightly-4.12-ocp-ovn-remote-libvirt-ppc64le	b4dc07d5dc48e31b07598bca18ba10d6c0d4d24d	link	unknown	`/pj-rehearse`
ci/rehearse/periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-heterogeneous	8fe905f8faac1ecc5bc1f89ed051ce10844da891	link	unknown	`/pj-rehearse`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

petr-muller · 2022-11-10T13:51:18Z

/hold cancel
/pj-rehearse

Updated OWNERS, also submitted #33932 separately so that I can help with maintaining all of these.

Can I have a re-/lgtm?

wking

/lgtm

openshift-ci · 2022-11-10T17:47:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aleskandro, deepsm007, petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/config/openshift/multiarch/OWNERS~~ [deepsm007]
~~ci-operator/jobs/openshift/multiarch/OWNERS~~ [deepsm007]
~~ci-operator/step-registry/ipi/install/heterogeneous/OWNERS~~ [aleskandro,deepsm007]
~~ci-operator/step-registry/openshift/e2e/aws/heterogeneous/OWNERS~~ [aleskandro,deepsm007,wking]
~~ci-operator/step-registry/openshift/upgrade/OWNERS~~ [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2022-11-10T18:12:28Z

@petr-muller: Updated the following 2 configmaps:

ci-operator-master-configs configmap in namespace ci at cluster app.ci using the following files:
- key openshift-multiarch-master__nightly-4.11.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.11.yaml
- key openshift-multiarch-master__nightly-4.12-upgrade-from-nightly-4.11.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.12-upgrade-from-nightly-4.11.yaml
- key openshift-multiarch-master__nightly-4.12-upgrade-from-stable-4.11.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.12-upgrade-from-stable-4.11.yaml
- key openshift-multiarch-master__nightly-4.12.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.12.yaml
- key openshift-multiarch-master__nightly-4.13-upgrade-from-nightly-4.12.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.13-upgrade-from-nightly-4.12.yaml
- key openshift-multiarch-master__nightly-4.13-upgrade-from-stable-4.12.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.13-upgrade-from-stable-4.12.yaml
- key openshift-multiarch-master__nightly-4.13.yaml using file ci-operator/config/openshift/multiarch/openshift-multiarch-master__nightly-4.13.yaml
job-config-master-periodics configmap in namespace ci at cluster app.ci using the following files:
- key openshift-multiarch-master-periodics.yaml using file ci-operator/jobs/openshift/multiarch/openshift-multiarch-master-periodics.yaml

Details

In response to this:

multiarch: add scaffolding for hetero->hetero upgrade jobs

make update

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2022

openshift-ci bot requested review from fabianofranz and jaypoulz November 4, 2022 13:25

petr-muller force-pushed the ota-656-hetero-updates branch from 25feab5 to 855afd9 Compare November 4, 2022 14:06

petr-muller force-pushed the ota-656-hetero-updates branch from 855afd9 to 86ab588 Compare November 4, 2022 14:13

petr-muller force-pushed the ota-656-hetero-updates branch from 86ab588 to f4c63fc Compare November 4, 2022 18:11

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 7, 2022

petr-muller force-pushed the ota-656-hetero-updates branch from f4c63fc to 7abc5d0 Compare November 7, 2022 16:13

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 7, 2022

petr-muller commented Nov 7, 2022

View reviewed changes

petr-muller force-pushed the ota-656-hetero-updates branch from 7abc5d0 to b4dc07d Compare November 7, 2022 16:18

petr-muller changed the title ~~WIP: OTA-656: Add CI jobs for heterogeneous cluster upgrades~~ OTA-656: Add CI jobs for heterogeneous cluster upgrades Nov 7, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2022

petr-muller force-pushed the ota-656-hetero-updates branch from b4dc07d to 8fe905f Compare November 7, 2022 16:35

openshift-ci bot requested review from aleskandro, deepsm007 and jottofar and removed request for fabianofranz and jaypoulz November 7, 2022 16:37

openshift-ci bot assigned deepsm007 Nov 8, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2022

aleskandro approved these changes Nov 8, 2022

View reviewed changes

openshift-ci bot assigned aleskandro Nov 8, 2022

openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Nov 9, 2022

petr-muller mentioned this pull request Nov 9, 2022

WIP: Add myself as upgrade CI content approver #33903

Closed

wking approved these changes Nov 9, 2022

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 9, 2022

openshift-ci bot assigned wking Nov 9, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 9, 2022

petr-muller added 3 commits November 10, 2022 14:35

multiarch: add scaffolding for hetero->hetero upgrade jobs

7c3e94f

heterogeneous: remove unnecessary code

6c0f784

- `$KUBECONFIG` is set by ci-operator - Console URL extraction only needs to be done by steps that *install* a cluster

make update

a9be5a1

petr-muller force-pushed the ota-656-hetero-updates branch from 9ce7d4e to 0debcc7 Compare November 10, 2022 13:35

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 10, 2022

petr-muller added 2 commits November 10, 2022 14:38

Introduce a shared heterogenenous upgrade workflow

60c71db

Improve heterogeneous install workflow description

287b13f

petr-muller force-pushed the ota-656-hetero-updates branch from 0debcc7 to 287b13f Compare November 10, 2022 13:39

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 10, 2022

petr-muller mentioned this pull request Nov 10, 2022

Wire heterogeneous upgrade jobs to multiarch release-controller #33934

Merged

wking approved these changes Nov 10, 2022

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 10, 2022

openshift-merge-robot merged commit ec16f10 into openshift:master Nov 10, 2022

OTA-656: Add CI jobs for heterogeneous cluster upgrades #33761

OTA-656: Add CI jobs for heterogeneous cluster upgrades #33761

Uh oh!

Conversation

petr-muller commented Nov 4, 2022

Uh oh!

openshift-ci-robot commented Nov 4, 2022

Uh oh!

petr-muller commented Nov 4, 2022

Uh oh!

petr-muller commented Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petr-muller commented Nov 4, 2022

Uh oh!

petr-muller commented Nov 4, 2022

Uh oh!

petr-muller commented Nov 4, 2022

Uh oh!

petr-muller commented Nov 4, 2022

Uh oh!

petr-muller commented Nov 7, 2022

Uh oh!

petr-muller Nov 7, 2022

Choose a reason for hiding this comment

Uh oh!

wking Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

deepsm007 Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

petr-muller commented Nov 7, 2022

Uh oh!

petr-muller commented Nov 7, 2022

Uh oh!

petr-muller commented Nov 7, 2022

Uh oh!

petr-muller commented Nov 9, 2022

Uh oh!

wking left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petr-muller commented Nov 10, 2022

Uh oh!

wking left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Nov 10, 2022

Uh oh!

openshift-ci bot commented Nov 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

petr-muller commented Nov 4, 2022 •

edited

Loading

openshift-ci bot commented Nov 10, 2022 •

edited

Loading