Add disaster recovery template

[vrutkovs]

Create a new template which would test cluster disaster recovery when 2 masters are down.

TODO:

• Use aws cli to remove masters - apparently Machine API won't let me destroy two masters for some reason. Seems to be an etcd issue

[sdodson]

Nice start, knowing that we'll have this and a cluster state rollback should we name the file/job so that it's clear this is specific to the infrastructure replacement DR scenario? Otherwise if it works as advertised I think we've completed the first step for the first scenario.

[vrutkovs]

Renamed to cluster-disaster-control-plane-ipi to mention which scenario and install type is being tested

[sdodson]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sdodson, vrutkovs
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: smarterclayton

If they are not already assigned, you can assign the PR to them by writing /assign @smarterclayton in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/templates/openshift/installer/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sdodson]

@wking or @staebler can we get an approve on this.

The goal here is to add a CI job that leads to quorum loss and then other teams will contribute tasks to restore the cluster. This job is not currently tied to any repo or scheduled periodically.

[wking]

nit with the filename, I think we can assume IPI and only call out UPI in UPI templates (which is our current pattern), so I'd prefer cluster-disaster-control-plane.yaml.

[openshift-ci-robot]

New changes are detected. LGTM label has been removed.

[sdodson]

Renamed and squashed down.

[wking]

One way to address this without a new template would be to drop all of this new stuff into a shell function, and jobs that wanted to trigger it could use:

- name: TEST_COMMAND
  value: |
    destroy-some-control-plane-machines
    TEST_SUITE=openshift/conformance/parallel run-tests

[patrickdillon]

When you say drop this stuff into a new shell function, are you saying the new shell function would be added to an already existing template, like cluster-launch-installer-e2e.yaml or another template? That makes sense, but I wanted to make sure that is your point.

[wking]

When you say drop this stuff into a new shell function, are you saying the new shell function would be added to an already existing template, like cluster-launch-installer-e2e.yaml or another template?

Yup, right here, with:

function destroy-some-control-plane-machines() {
  ...bla bla...
}

[vrutkovs]

Created #3572 to play with this idea

[vrutkovs]

Closed in favor of #3572