ci-operator/config/openshift/release: Drop 4.8 -> 4.9 -> 4.8 rollback jobs

[wking]

4.9 includes the backwards-incompatible etcd disk-schema change from etcd v3.5.0. That causes rollback jobs to fail like:

Working towards 4.8.12: 69 of 678 done (10% complete)

where the cluster-version operator is waiting for the etcd operator (but hasn't been waiting quite long enough to complain about it by name):

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade-rollback/1441850131237310464/artifacts/e2e-aws-ovn-upgrade-rollback/gather-extra/artifacts/pods/openshift-cluster-version_cluster-version-operator-7cfbc65959-xctxv_cluster-version-operator.log | grep 'Running sync.*in state\|Result of work' | tail -n3
I0925 23:25:54.121866       1 sync_worker.go:541] Running sync registry.build01.ci.openshift.org/ci-op-sqh94dxj/release@sha256:c3af995af7ee85e88c43c943e0a64c7066d90e77fafdabc7b22a095e4ea3c25a (force=true) on generation 3 in state Updating at attempt 11
I0925 23:31:36.036075       1 task_graph.go:555] Result of work: [Cluster operator etcd is degraded]
I0925 23:34:38.987554       1 sync_worker.go:541] Running sync registry.build01.ci.openshift.org/ci-op-sqh94dxj/release@sha256:c3af995af7ee85e88c43c943e0a64c7066d90e77fafdabc7b22a095e4ea3c25a (force=true) on generation 3 in state Updating at attempt 12

Seeing what the etcd operator has to say:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade-rollback/1441850131237310464/artifacts/e2e-aws-ovn-upgrade-rollback/gather-extra/artifacts/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "etcd").status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")'
2021-09-25T19:56:27Z RecentBackup=Unknown ControllerStarted: -
2021-09-25T22:01:04Z Degraded=True EtcdMembers_UnhealthyMembers::StaticPods_Error: EtcdMembersDegraded: 2 of 3 members are available, ip-10-0-246-48.us-east-2.compute.internal is unhealthy
StaticPodsDegraded: pod/etcd-ip-10-0-246-48.us-east-2.compute.internal container "etcd" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-ip-10-0-246-48.us-east-2.compute.internal_openshift-etcd(967f9e83-e6a2-437e-85e6-c33563286f7f)
2021-09-25T21:55:07Z Progressing=True NodeInstaller: NodeInstallerProgressing: 3 nodes are at revision 4; 0 nodes have achieved new revision 5
2021-09-25T19:57:59Z Available=True AsExpected: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4; 0 nodes have achieved new revision 5
EtcdMembersAvailable: 2 of 3 members are available, ip-10-0-246-48.us-east-2.compute.internal is unhealthy
2021-09-25T19:56:43Z Upgradeable=True AsExpected: All is well

And from the logs of that container:

{"level":"fatal","ts":"2021-09-25T23:34:47.679Z","caller":"membership/cluster.go:790","msg":"invalid downgrade; server version is lower than determined cluster version","current-server-version":"3.4.14","determined-cluster-version":"3.5","stacktrace":"go.etcd.io/etcd/etcdserver/api/membership.mustDetectDowngrade\n\t/go/src/go.etcd.io/etcd/etcdserver/api/membership/cluster.go:790...

We know these rollback jobs will always stick on that etcd disk-schema rollback, so no sense in spending CI money running them and watching them fail. This commit drops the jobs, and we won't worry about other minor-rollback issues around 4.8-to-4.9.

[stbenjam]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stbenjam, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/release/OWNERS [stbenjam]
• ci-operator/jobs/openshift/release/OWNERS [stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@wking: Updated the following 2 configmaps:

• ci-operator-master-configs configmap in namespace ci at cluster app.ci using the following files:
  • key openshift-release-master__ci-4.9-upgrade-from-stable-4.8.yaml using file ci-operator/config/openshift/release/openshift-release-master__ci-4.9-upgrade-from-stable-4.8.yaml
• job-config-master configmap in namespace ci at cluster app.ci using the following files:
  • key openshift-release-master-periodics.yaml using file ci-operator/jobs/openshift/release/openshift-release-master-periodics.yaml

Details

In response to this:

4.9 includes the backwards-incompatible etcd disk-schema change from etcd v3.5.0. That causes rollback jobs to fail like:

Working towards 4.8.12: 69 of 678 done (10% complete)

where the cluster-version operator is waiting for the etcd operator (but hasn't been waiting quite long enough to complain about it by name):

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade-rollback/1441850131237310464/artifacts/e2e-aws-ovn-upgrade-rollback/gather-extra/artifacts/pods/openshift-cluster-version_cluster-version-operator-7cfbc65959-xctxv_cluster-version-operator.log | grep 'Running sync.*in state\|Result of work' | tail -n3
I0925 23:25:54.121866       1 sync_worker.go:541] Running sync registry.build01.ci.openshift.org/ci-op-sqh94dxj/release@sha256:c3af995af7ee85e88c43c943e0a64c7066d90e77fafdabc7b22a095e4ea3c25a (force=true) on generation 3 in state Updating at attempt 11
I0925 23:31:36.036075       1 task_graph.go:555] Result of work: [Cluster operator etcd is degraded]
I0925 23:34:38.987554       1 sync_worker.go:541] Running sync registry.build01.ci.openshift.org/ci-op-sqh94dxj/release@sha256:c3af995af7ee85e88c43c943e0a64c7066d90e77fafdabc7b22a095e4ea3c25a (force=true) on generation 3 in state Updating at attempt 12

Seeing what the etcd operator has to say:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade-rollback/1441850131237310464/artifacts/e2e-aws-ovn-upgrade-rollback/gather-extra/artifacts/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "etcd").status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")'
2021-09-25T19:56:27Z RecentBackup=Unknown ControllerStarted: -
2021-09-25T22:01:04Z Degraded=True EtcdMembers_UnhealthyMembers::StaticPods_Error: EtcdMembersDegraded: 2 of 3 members are available, ip-10-0-246-48.us-east-2.compute.internal is unhealthy
StaticPodsDegraded: pod/etcd-ip-10-0-246-48.us-east-2.compute.internal container "etcd" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd pod=etcd-ip-10-0-246-48.us-east-2.compute.internal_openshift-etcd(967f9e83-e6a2-437e-85e6-c33563286f7f)
2021-09-25T21:55:07Z Progressing=True NodeInstaller: NodeInstallerProgressing: 3 nodes are at revision 4; 0 nodes have achieved new revision 5
2021-09-25T19:57:59Z Available=True AsExpected: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4; 0 nodes have achieved new revision 5
EtcdMembersAvailable: 2 of 3 members are available, ip-10-0-246-48.us-east-2.compute.internal is unhealthy
2021-09-25T19:56:43Z Upgradeable=True AsExpected: All is well

And from the logs of that container:

{"level":"fatal","ts":"2021-09-25T23:34:47.679Z","caller":"membership/cluster.go:790","msg":"invalid downgrade; server version is lower than determined cluster version","current-server-version":"3.4.14","determined-cluster-version":"3.5","stacktrace":"go.etcd.io/etcd/etcdserver/api/membership.mustDetectDowngrade\n\t/go/src/go.etcd.io/etcd/etcdserver/api/membership/cluster.go:790...

We know these rollback jobs will always stick on that etcd disk-schema rollback, so no sense in spending CI money running them and watching them fail. This commit drops the jobs, and we won't worry about other minor-rollback issues around 4.8-to-4.9.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.