Bug 1796627: cmd/openshift-install/create: add env var to preserve bootstrap #2935

stbenjam · 2020-01-16T14:22:51Z

For a variety of reasons, it may be useful to keep the bootstrap VM
around for debugging. My general approach has been to interrupt the
installer before it has a chance to delete the bootstrap VM.

If the environment variable OPENSHIFT_INSTALL_PRESERVE_BOOTSTRAP is set
to any value, the installer will not delete the bootstrap.

williamcaban · 2020-01-16T14:31:32Z

We need this option for troubleshooting failure on disconnected environments where we are unable to further troubleshoot failed deployment because the bootstrap is not longer there.

abhinavdahiya · 2020-01-17T10:52:26Z

Hmm if the cluster bootstrap succeeds I don't see how keeping the bootstrap is useful?
If the cluster bootstrap doesn't complete the bootstrap is not destroyed.

stbenjam · 2020-01-17T11:13:00Z

I have had many occasions where bootstrap succeeds, but installation failed and it would be useful to examine the bootstrap node. Yesterday we had an issue with machine-config-operator rendering different manifests on bootstrap vs the cluster but the bootstrap had already been removed so we couldn’t examine why.

sdodson · 2020-01-20T15:29:55Z

#1692 previously closed PR in the same direction

I think #2936 is a more generally applicable solution because it narrows the gap between bootstrap and running cluster state.

sdodson · 2020-01-24T20:26:04Z

@abhinavdahiya would you be in support of this being available as an installer build time option? (never to be represented in product docs)

sdodson · 2020-01-29T18:58:37Z

@stbenjam

We've discussed the need for this and have arrived at the conclusion that we'd accept an environment variable that preserved bootstrap host and emitted warnings advising that this is intended only for debugging purposes and poses a risk to cluster stability. Similar to the warning emitted when the release image is overridden via env var.

Would you be interested in updating your work for that or would you prefer someone from Installer team take that on?

stbenjam · 2020-01-29T19:02:01Z

@stbenjam

We've discussed the need for this and have arrived at the conclusion that we'd accept an environment variable that preserved bootstrap host and emitted warnings advising that this is intended only for debugging purposes and poses a risk to cluster stability. Similar to the warning emitted when the release image is overridden via env var.

Would you be interested in updating your work for that or would you prefer someone from Installer team take that on?

Sure, thanks - I'll update the PR this afternoon.

For a variety of reasons, it may be useful to keep the bootstrap VM around for debugging. My general approach has been to interrupt the installer before it has a chance to delete the bootstrap VM. If the environment variable OPENSHIFT_INSTALL_PRESERVE_BOOTSTRAP is set to any value, the installer will not delete the bootstrap.

stbenjam · 2020-01-29T19:15:38Z

@sdodson PTAL

sdodson · 2020-01-29T19:24:20Z

@stbenjam Thanks, while we don't intend to allow this to enter into CI jobs because we don't wish for CI to diverge from normal customer workflows, we will likely enable this via clusterbot. Does openshift-install destroy cluster destroy the bootstrap host if it's left intact by this change? If not we need to at least get an issue in to track adding that.

sdodson · 2020-01-29T19:24:47Z

/approve
@patrickdillon PTAL

openshift-ci-robot · 2020-01-29T19:25:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

stbenjam · 2020-01-29T19:27:17Z

Does openshift-install destroy cluster destroy the bootstrap host if it's left intact by this change?

I believe so, any terraform resources that still exist should get destroyed - but I haven't tested yet.

sdodson · 2020-01-30T13:49:51Z

/lgtm

openshift-bot · 2020-01-30T14:23:28Z

/retest