installer template: add 'recover from etcd quorum loss' test

[vrutkovs]

Rework of #3495 - run etcd quorum loss as a dedicated test in installer template.

This adds a new optional test e2e-etcd-quorum-loss which simulates the quorum loss. At this point its expected to fail as a single etcd node doesn't get restored yet.

This PR uses a forked version of scripts served by MCO. This would be fixed in a separate PR so that we could work in parallel on various small tasks related to DR

TODO:

• Rebase when installer template: add a function to test cluster state restore #3595 lands

[vrutkovs]

Failing tests: 
[Feature:Platform][Smoke] Managed cluster should ensure control plane operators do not make themselves unevictable [Suite:openshift/conformance/parallel] 
[Feature:Platform][Smoke] Managed cluster should ensure pods use images from our release image with proper ImagePullPolicy [Suite:openshift/conformance/parallel]

Fixed in the latest commit - remove ssh-bastion, scale etcd-quorum-guard to 3 and remove etcd-signer pod

[wking]

Looks good. Left a few nits inline. It's not clear to me how much of this code is staying vs. moving out the the machine-config operator. Is it just the here-docs moving out? If so, there's a bit of shared functionality between this and restore-cluster-state that could be pulled out into helper functions to stay DRY.

[wking]

What are we waiting for here? This seems brittle.

[vrutkovs]

Machine API won't wait for machine to be deleted during oc delete machine call, so API is available for a few seconds after 2 masters were removed.

This pause is necessary to confirm API is no longer responding

[wking]

floating downloads 😭 😉

[vrutkovs]

Its a temporary measure while we debug MCO scripts.

This is fixed in #3842 which is the same PR + MCO scripts

[wking]

Not part of the test framework that I can review, but if your etcd cluster is down to one node (because you killed two of three control-plane nodes), then how is the remaining etcd still functioning? I'd have expected it to be freaking out about having lost quorum and refusing to take possibly-split-brained actions. If there's some quick explanation for how this works, I'm very curious. If the explanation is longer, we should probably skip it to avoid distracting from the test script itself.

[vrutkovs]

if your etcd cluster is down to one node (because you killed two of three control-plane nodes), then how is the remaining etcd still functioning?

etcd-snapshot-restore.sh without params would restore this etcd forming a one node cluster - https://github.com/openshift/machine-config-operator/blob/master/templates/master/00-master/_base/files/usr-local-bin-openshift-recovery-tools-sh.yaml#L141-L168.

Added a bit better comment for this in b99911295

[vrutkovs]

Fixed all outstanding issues, rebased on master.

This would eventually be substituted by #3842, changes to use MCO are contained in ad5a69f.

Some scripts would land in origin tests image - see openshift/origin#22887

[vrutkovs]

fail [github.com/openshift/origin/test/extended/operators/cluster.go:118]: Expected
    <[]string | len:4, cap:4>: [
        "Pod openshift-apiserver-operator/openshift-apiserver-operator-56c559db6f-xfmhm is not healthy: container openshift-apiserver-operator has restarted more than 5 times",
        "Pod openshift-controller-manager-operator/openshift-controller-manager-operator-85cc57f9d7-qvgnm is not healthy: container operator has restarted more than 5 times",
        "Pod openshift-kube-apiserver-operator/kube-apiserver-operator-84c9f5b76c-rklfj is not healthy: container kube-apiserver-operator has restarted more than 5 times",
        "Pod openshift-kube-scheduler-operator/openshift-kube-scheduler-operator-6c677b65cc-rmw7r is not healthy: container kube-scheduler-operator-container has restarted more than 5 times",
    ]
to be empty

/test pj-rehearse

[openshift-ci-robot]

@vrutkovs: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/rehearse/openshift/installer/master/e2e-vsphere	75352c6446e2ffae369b56381c914245bf0aa37c	link	/test pj-rehearse
ci/rehearse/openshift/builder/master/e2e-aws	9db3c7f	link	/test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-etcd-quorum-loss	9db3c7f	link	/test pj-rehearse
ci/prow/pj-rehearse	9db3c7f	link	/test pj-rehearse

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[vrutkovs]

Superceded by #3842