Bug 1791993: proxy: use explicit list of platforms for metadata addresses #2939

stbenjam · 2020-01-16T20:09:35Z

The installer creates a manifest for proxy configuration, automatically
adding specific addresses to NO_PROXY depending on the platform. One of
those addresses is the metadata service, hosted at 169.254.169.254. The
installer assumes this must be done for all platforms other than None or
vSphere, whereas the cluster-network-operator has an explicit list of
platforms:

https://github.com/openshift/cluster-network-operator/blob/adaf257b4d63661726443ab2b059a9b4209a02d1/pkg/util/proxyconfig/no_proxy.go#L67-L69

When using a proxy with baremetal IPI, the installer adds this address,
however when the CNO comes up, it does not, causing the rendered
machine configs to differ, and installation to fail, with MCO reporting
errors like:

pool master has not progressed to latest configuration: configuration
status for pool master is empty: pool is degraded because nodes fail
with "3 nodes are reporting degraded status on sync": "Node master-1 is
reporting: \"machineconfig.machineconfiguration.openshift.io
\\\"rendered-master-982b8698753da7e31b5f902aa4dc135e\\\" not found\""

This needs a better, longer term solution to ensure the installer and
CNO are not creating conflicting proxy objects, however as a short-term
fix that is easily backportable to 4.3 to ensure proxies work on
baremetal, this syncs the two lists between the installer and CNO.

The installer creates a manifest for proxy configuration, automatically adding specific addresses to NO_PROXY depending on the platform. One of those addresses is the metadata service, hosted at 169.254.169.254. The installer assumes this must be done for all platforms other than None of vSphere, whereas the cluster-network-operator has an explicit list of platforms: https://github.com/openshift/cluster-network-operator/blob/adaf257b4d63661726443ab2b059a9b4209a02d1/pkg/util/proxyconfig/no_proxy.go#L67-L69 When using a proxy with baremetal IPI, the installer adds this address, however when the CNO comes up, it does not, causing the rendered machine configs to differ, and installation to fail, with MCO reporting errors like: ``` pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with "3 nodes are reporting degraded status on sync": "Node master-1 is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-982b8698753da7e31b5f902aa4dc135e\\\" not found\"" ``` This needs a better, longer term solution to ensure the installer and CNO are not creating conflicting proxy objects, however as a short-term fix that is easily backportable to 4.3 to ensure proxies work on baremetal, this syncs the two lists between the installer and CNO.

cgwalters · 2020-01-16T20:13:43Z

Ahh, that's the bug behind openshift/machine-config-operator#1376 ?

It feels like the cleanest fix here is to add the CNO into the bootstrap phase explicitly, rather than duplicate what it's doing in Terratform?

cgwalters · 2020-01-16T20:16:38Z

Excellent commit message (as usual) BTW!

And just for completeness here's the exact code in the CNO: https://github.com/openshift/cluster-network-operator/blob/adaf257b4d63661726443ab2b059a9b4209a02d1/pkg/util/proxyconfig/no_proxy.go#L67

stbenjam · 2020-01-16T20:16:52Z

Ahh, that's the bug behind openshift/machine-config-operator#1376 ?

It feels like the cleanest fix here is to add the CNO into the bootstrap phase explicitly, rather than duplicate what it's doing in Terratform?

That sounds like the right solution, however given that this affects 4.3, and we have some users affected by this, do you think there's a chance we could sync the two lists of platforms for now, and then work on a longer term fix?

cgwalters · 2020-01-16T20:16:52Z

/approve

stbenjam · 2020-01-16T20:17:07Z

/cherry-pick release-4.3

openshift-cherrypick-robot · 2020-01-16T20:17:08Z

@stbenjam: once the present PR merges, I will cherry-pick it on top of release-4.3 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2020-01-16T20:17:35Z

do you think there's a chance we could sync the two lists of platforms for now, and then work on a longer term fix?

Yep, I 100% agree with landing this fix as is now.

openshift-ci-robot · 2020-01-16T20:19:46Z

@stbenjam: This pull request references Bugzilla bug 1791993, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1791993: proxy: use explicit list of platforms for metadata addresses

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stbenjam · 2020-01-16T20:19:51Z

4.3 BZ is 1791995

stbenjam · 2020-01-16T20:20:14Z

/bugzilla refresh

openshift-ci-robot · 2020-01-16T20:20:17Z

@stbenjam: This pull request references Bugzilla bug 1791993, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stbenjam · 2020-01-16T20:20:58Z

I didn't hit save 🤦‍♂️

/bugzilla refresh

openshift-ci-robot · 2020-01-16T20:21:03Z

@stbenjam: This pull request references Bugzilla bug 1791993, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

I didn't hit save 🤦‍♂️

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2020-01-16T20:39:03Z

/lgtm

wking · 2020-01-16T20:41:26Z

/approve

Maybe we need explicit approval now?

openshift-ci-robot · 2020-01-16T20:42:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

stbenjam · 2020-01-16T21:20:26Z

/test e2e-aws-upgrade

openshift-bot · 2020-01-16T21:29:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-16T21:42:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-16T23:00:20Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-16T23:39:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-16T23:53:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T01:10:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T01:23:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T02:15:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T02:29:25Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T03:07:42Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T03:20:57Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T04:25:43Z

/retest

Please review the full test history for this PR and help us cut down flakes.

abhinavdahiya · 2020-01-17T10:30:14Z

Hmm there are more clouds than none/baremetal so this metadata ip being added to default and none/baremetal opting out seems like a maintainable solution imo.

stbenjam · 2020-01-17T12:21:45Z

Hmm there are more clouds than none/baremetal so this metadata ip being added to default and none/baremetal opting out seems like a maintainable solution imo.

I am fine with either approach, but this change has the benefit of only requiring 1 change instead of 2.

What did you think of @cgwalters suggestion? IMHO I agree the best approach is not having the same thing done in two different codebases and avoid introducing this risk of conflict at all. #2939 (comment)

openshift-bot · 2020-01-17T14:29:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T14:37:42Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T14:56:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T15:55:15Z

/retest

Please review the full test history for this PR and help us cut down flakes.

stbenjam · 2020-01-17T17:58:52Z

/test e2e-aws

openshift-bot · 2020-01-17T18:31:41Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T18:44:13Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-17T19:10:31Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-01-17T19:28:24Z

@stbenjam: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-ovirt	`f92b14d`	link	`/test e2e-ovirt`
ci/prow/e2e-openstack	`f92b14d`	link	`/test e2e-openstack`
ci/prow/e2e-libvirt	`f92b14d`	link	`/test e2e-libvirt`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2020-01-17T21:16:12Z

@stbenjam: All pull requests linked via external trackers have merged. Bugzilla bug 1791993 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1791993: proxy: use explicit list of platforms for metadata addresses

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2020-01-17T21:16:34Z

@stbenjam: new pull request created: #2944

Details

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-01T01:36:11Z

@stbenjam: Bugzilla bug 1791993 is in an unrecognized state (ON_QA) and will not be moved to the MODIFIED state.

Details

In response to this:

Bug 1791993: proxy: use explicit list of platforms for metadata addresses

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 16, 2020

openshift-ci-robot requested review from fabianofranz and patrickdillon January 16, 2020 20:10

stbenjam changed the title ~~proxy: use explicit list of platforms for metadata addresses~~ Bug 1791993: proxy: use explicit list of platforms for metadata addresses Jan 16, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jan 16, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jan 16, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jan 16, 2020

openshift-ci-robot assigned wking Jan 16, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2020

stbenjam mentioned this pull request Jan 17, 2020

baremetal: http_proxy prevents installer-ironic API calls #2904

Closed

openshift-merge-robot merged commit ae1de17 into openshift:master Jan 17, 2020

openshift-cherrypick-robot mentioned this pull request Jan 17, 2020

[release-4.3] Bug 1791995: proxy: use explicit list of platforms for metadata addresses #2944

Merged

stbenjam deleted the proxy branch January 17, 2020 21:21

cgwalters mentioned this pull request Feb 1, 2020

Phase out special bootstrap machine-config components openshift/machine-config-operator#1427

Closed

stbenjam mentioned this pull request May 24, 2021

Add the metal3-state host to NoProxy openshift/cluster-network-operator#1106

Closed

Bug 1791993: proxy: use explicit list of platforms for metadata addresses #2939

Bug 1791993: proxy: use explicit list of platforms for metadata addresses #2939

Uh oh!

Conversation

stbenjam commented Jan 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Jan 16, 2020

Uh oh!

cgwalters commented Jan 16, 2020

Uh oh!

stbenjam commented Jan 16, 2020

Uh oh!

cgwalters commented Jan 16, 2020

Uh oh!

stbenjam commented Jan 16, 2020

Uh oh!

openshift-cherrypick-robot commented Jan 16, 2020

Uh oh!

cgwalters commented Jan 16, 2020

Uh oh!

openshift-ci-robot commented Jan 16, 2020

Uh oh!

stbenjam commented Jan 16, 2020

Uh oh!

stbenjam commented Jan 16, 2020

Uh oh!

openshift-ci-robot commented Jan 16, 2020

Uh oh!

stbenjam commented Jan 16, 2020

Uh oh!

openshift-ci-robot commented Jan 16, 2020

Uh oh!

wking commented Jan 16, 2020

Uh oh!

wking commented Jan 16, 2020

Uh oh!

openshift-ci-robot commented Jan 16, 2020

Uh oh!

stbenjam commented Jan 16, 2020

Uh oh!

openshift-bot commented Jan 16, 2020

Uh oh!

openshift-bot commented Jan 16, 2020

Uh oh!

openshift-bot commented Jan 16, 2020

Uh oh!

openshift-bot commented Jan 16, 2020

Uh oh!

openshift-bot commented Jan 16, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

abhinavdahiya commented Jan 17, 2020

Uh oh!

stbenjam commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

Uh oh!

stbenjam commented Jan 17, 2020

Uh oh!

openshift-bot commented Jan 17, 2020

stbenjam commented Jan 16, 2020 •

edited

Loading

openshift-ci-robot commented Jan 17, 2020 •

edited

Loading