Bug 2072202: Check for api and api-int resolution during cluster install #5816

sadasu · 2022-04-14T20:50:17Z

During cluster bring up, when nodes are unable to resolve api-int, the cluster fails to come up and the messages in the logs, several times do not hint at any connectivity issues making it difficult to detect and resolve any infrastructure setup issues.

These changes contain:

Checks for the API and API-INT URLs to see if they are resolvable from the bootstrap node and logging helpful messages.
Records the state of the api-url and api-int-url services
Addition of validation of the API and API-INT URLs to the analyze output

sadasu · 2022-04-18T15:42:11Z

/retest

patrickdillon

It is unlikely that api-int is resolvable from the installer host, so this check would need to be run on the bootstrap host.

I think it would be a good idea to enumerate the failure cases we are trying to improve upon.

jcpowermac · 2022-04-20T17:42:45Z

It is unlikely that api-int is resolvable from the installer host, so this check would need to be run on the bootstrap host.

right, its not. the coredns static pod running on the nodes configures the A record for api-int

see:
https://github.com/openshift/machine-config-operator/blob/master/templates/common/on-prem/files/coredns-corefile.yaml#L32-L39

and
https://github.com/openshift/machine-config-operator/blob/master/manifests/on-prem/coredns-corefile.tmpl#L27-L35

patrickdillon · 2022-04-20T17:55:06Z

right, its not.

The installer would be able to resolve api-int in the edge case where it is installing to an existing vpc (e.g. aws, azure, gcp) and the host is running within that vpc, right?

Like I said, this is an edge case and does not affect the outcome for this PR, but trying to add context for @sadasu about why I said "unlikely." Also trying to edify myself in case I'm mistaken.

jcpowermac · 2022-04-20T17:56:56Z

right, its not.

The installer would be able to resolve api-int in the edge case where it is installing to an existing vpc (e.g. aws, azure, gcp) and the host is running within that vpc, right?

Like I said, this is an edge case and does not affect the outcome for this PR, but trying to add context for @sadasu about why I said "unlikely." Also trying to edify myself in case I'm mistaken.

Sorry yep, I was thinking onprem ;-)

sadasu · 2022-04-29T22:14:26Z

@patrickdillon @jcpowermac thanks for the helpful comments. Hopefully the current location triggers this verification at the correct point on the bootstrap node on an install failure.

Couple of questions:

Is *.apps expected to be resolvable at this point? Should these changes include a test to see if that URL is resolvable too?
Should this check also include a ping to the API or API-INT IP address?

jstuever · 2022-05-03T17:47:04Z

/cc

jstuever

This looks like a solid approach. I do have questions about how the new shell script is being referenced from bootkube.sh. I expect this is missing a source line at the top. Otherwise, looks good.

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

patrickdillon

This looks good. Left some nits and ideas. Personally I would like to consider some different terminology than "DNS config."

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

data/data/bootstrap/files/usr/local/bin/bootstrap-verify-dns-config.sh

pkg/gather/service/analyze.go

sadasu · 2022-10-08T14:47:16Z

/retest-required

sadasu · 2022-10-10T15:32:40Z

/retest-required

r4f4 · 2022-10-10T18:58:31Z

/test e2e-alibaba

r4f4

Except for some nitpicking on the indentation of some lines, it seems to be working as intended:

level=error msg=Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.
level=error msg=The bootstrap machine is unable to resolve API and/or API-Int Server URLs
level=info msg=Successfully resolved API_INT_URL api-int.ci-op-97x1j7tm-920ba.alicloud-dev.devcluster.openshift.com
level=info msg=Unable to reach API_INT_URL's https endpoint at https://10.0.101.111:6443/version
level=info msg=It might be too early for the https://10.0.101.111:6443/version to be available.

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/5816/pull-ci-openshift-installer-master-e2e-alibaba/1579546984006553600

data/data/bootstrap/files/usr/local/bin/bootstrap-verify-api-server-urls.sh

On the bootstrap node, verify if the API URL and API-INT URLS are resolvable when cluster bringup fails.

r4f4 · 2022-10-11T13:15:58Z

/lgtm

openshift-ci-robot · 2022-10-11T18:39:28Z

/retest-required

Remaining retests: 0 against base HEAD 54d95a3 and 2 for PR HEAD 6bfb2e6 in total

sadasu · 2022-10-11T21:04:11Z

/retest-required

openshift-ci-robot · 2022-10-12T00:57:36Z

/retest-required

Remaining retests: 0 against base HEAD 3872845 and 1 for PR HEAD 6bfb2e6 in total

openshift-ci-robot · 2022-10-12T10:57:20Z

/retest-required

Remaining retests: 0 against base HEAD f957eda and 0 for PR HEAD 6bfb2e6 in total

sadasu · 2022-10-12T13:54:35Z

/skip
There is enough proof with the passing CIs that this fix works and doesn't break anything else.

jstuever · 2022-10-12T16:06:58Z

/cc @patrickdillon
/uncc

openshift-ci-robot · 2022-10-12T16:57:30Z

/hold

Revision 6bfb2e6 was retested 3 times: holding

openshift-ci · 2022-10-12T17:33:25Z

@sadasu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-crc	b4de9f26fa2e7fc90a6babcb91f81caab819434c	link	false	`/test e2e-crc`
ci/prow/e2e-gcp-upi	b4de9f26fa2e7fc90a6babcb91f81caab819434c	link	true	`/test e2e-gcp-upi`
ci/prow/e2e-aws-upi	b4de9f26fa2e7fc90a6babcb91f81caab819434c	link	true	`/test e2e-aws-upi`
ci/prow/e2e-azure-upi	b4de9f26fa2e7fc90a6babcb91f81caab819434c	link	true	`/test e2e-azure-upi`
ci/prow/okd-e2e-aws	0ca8925920bd26a79bc17f1a7607004f2d8bd21b	link	false	`/test okd-e2e-aws`
ci/prow/e2e-azure	b93e83b5e7693c4013815c0721afbb48b3405b25	link	true	`/test e2e-azure`
ci/prow/e2e-vsphere	b93e83b5e7693c4013815c0721afbb48b3405b25	link	true	`/test e2e-vsphere`
ci/prow/e2e-ibmcloud	b93e83b5e7693c4013815c0721afbb48b3405b25	link	false	`/test e2e-ibmcloud`
ci/prow/e2e-aws	b93e83b5e7693c4013815c0721afbb48b3405b25	link	true	`/test e2e-aws`
ci/prow/e2e-gcp	b93e83b5e7693c4013815c0721afbb48b3405b25	link	true	`/test e2e-gcp`
ci/prow/okd-e2e-gcp-ovn-upgrade	f72166fe50e34c5912dcaa9d496050e927176167	link	false	`/test okd-e2e-gcp-ovn-upgrade`
ci/prow/okd-scos-e2e-aws-ovn	`6bfb2e6`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/okd-scos-e2e-aws-upgrade	`6bfb2e6`	link	false	`/test okd-scos-e2e-aws-upgrade`
ci/prow/okd-e2e-aws-upgrade	`6bfb2e6`	link	false	`/test okd-e2e-aws-upgrade`
ci/prow/e2e-openstack	`6bfb2e6`	link	false	`/test e2e-openstack`
ci/prow/e2e-ibmcloud-ovn	`6bfb2e6`	link	false	`/test e2e-ibmcloud-ovn`
ci/prow/e2e-alibaba	a8569dfc04922c2191113dad7f4c99eec9c36a5c	link	false	`/test e2e-alibaba`
ci/prow/e2e-libvirt	`6bfb2e6`	link	false	`/test e2e-libvirt`
ci/prow/e2e-agent-mce	`6bfb2e6`	link	false	`/test e2e-agent-mce`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sadasu · 2022-10-12T18:50:33Z

/test e2e-gcp-ovn

sadasu · 2022-10-13T13:42:33Z

/hold cancel

openshift-ci-robot · 2022-10-13T15:34:23Z

/retest-required

Remaining retests: 0 against base HEAD 36ebaee and 2 for PR HEAD 6bfb2e6 in total

openshift-ci · 2022-10-13T18:26:38Z

@sadasu: All pull requests linked via external trackers have merged:

openshift/installer#5816

Bugzilla bug 2072202 has been moved to the MODIFIED state.

Details

In response to this:

Bug 2072202: Check for api and api-int resolution during cluster install

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 14, 2022

openshift-ci bot requested review from AnnaZivkovic and jhixson74 April 14, 2022 20:51

sadasu force-pushed the add-install-failure-check branch 2 times, most recently from e8e03e9 to 891fe87 Compare April 15, 2022 16:58

sadasu changed the title ~~WIP: Check for api-int resolution when cluster install fails~~ Check for api-int resolution when cluster install fails Apr 15, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 15, 2022

patrickdillon reviewed Apr 18, 2022

View reviewed changes

sadasu force-pushed the add-install-failure-check branch from 891fe87 to 24bcc65 Compare April 29, 2022 22:08

sadasu force-pushed the add-install-failure-check branch 2 times, most recently from 5828d56 to 230e2e0 Compare April 29, 2022 22:27

openshift-ci bot requested a review from jstuever May 3, 2022 17:47

jstuever requested changes May 3, 2022

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

sadasu force-pushed the add-install-failure-check branch 6 times, most recently from 2050020 to 64d16cd Compare May 4, 2022 19:05

patrickdillon reviewed May 5, 2022

View reviewed changes

sadasu force-pushed the add-install-failure-check branch 4 times, most recently from 65307c8 to 84928d0 Compare May 5, 2022 22:05

sadasu force-pushed the add-install-failure-check branch 2 times, most recently from 472c869 to a8569df Compare October 10, 2022 17:06

r4f4 reviewed Oct 10, 2022

View reviewed changes

sadasu added 2 commits October 10, 2022 19:21

Verify if api and api-int are resolvable when cluster bringup fails

149173a

On the bootstrap node, verify if the API URL and API-INT URLS are resolvable when cluster bringup fails.

Add the API and API-INT URL checks to the analyze output

6bfb2e6

sadasu force-pushed the add-install-failure-check branch from a8569df to 6bfb2e6 Compare October 10, 2022 23:25

openshift-ci bot assigned r4f4 Oct 11, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 11, 2022

openshift-ci bot requested review from patrickdillon and removed request for jstuever October 12, 2022 16:07

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 12, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 13, 2022

openshift-merge-robot merged commit 3d0aac7 into openshift:master Oct 13, 2022

openshift-ci bot mentioned this pull request Jan 26, 2023

Bug 2072202: Check for reachability of API and API-Int URLs later in bootkube #6611

Merged

Bug 2072202: Check for api and api-int resolution during cluster install #5816

Bug 2072202: Check for api and api-int resolution during cluster install #5816

Uh oh!

Conversation

sadasu commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sadasu commented Apr 18, 2022

Uh oh!

patrickdillon left a comment

Choose a reason for hiding this comment

Uh oh!

jcpowermac commented Apr 20, 2022

Uh oh!

patrickdillon commented Apr 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcpowermac commented Apr 20, 2022

Uh oh!

sadasu commented Apr 29, 2022

Uh oh!

jstuever commented May 3, 2022

Uh oh!

jstuever left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patrickdillon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sadasu commented Oct 8, 2022

Uh oh!

sadasu commented Oct 10, 2022

Uh oh!

r4f4 commented Oct 10, 2022

Uh oh!

r4f4 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

r4f4 commented Oct 11, 2022

Uh oh!

openshift-ci-robot commented Oct 11, 2022

Uh oh!

sadasu commented Oct 11, 2022

Uh oh!

openshift-ci-robot commented Oct 12, 2022

Uh oh!

openshift-ci-robot commented Oct 12, 2022

Uh oh!

sadasu commented Oct 12, 2022

Uh oh!

jstuever commented Oct 12, 2022

Uh oh!

openshift-ci-robot commented Oct 12, 2022

Uh oh!

openshift-ci bot commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sadasu commented Oct 12, 2022

Uh oh!

sadasu commented Oct 13, 2022

Uh oh!

openshift-ci-robot commented Oct 13, 2022

Uh oh!

openshift-ci bot commented Oct 13, 2022

Uh oh!

Reviewers

Assignees

sadasu commented Apr 14, 2022 •

edited

Loading

patrickdillon commented Apr 20, 2022 •

edited

Loading

r4f4 left a comment •

edited

Loading

openshift-ci bot commented Oct 12, 2022 •

edited

Loading