controller-manager/scheduler: switch to secure ports by s-urbaniak · Pull Request #1576 · openshift/installer

s-urbaniak · 2019-04-10T09:54:02Z

Recently, the control plane switched to secure ports in [1] and [2].
This aligns them in the installer.

[1]
openshift/cluster-kube-scheduler-operator#88
[2]
openshift/cluster-kube-controller-manager-operator#207

Recently, the control plane switched to secure ports in [1] and [2]. This aligns them in the installer. [1] openshift/cluster-kube-scheduler-operator#88 [2] openshift/cluster-kube-controller-manager-operator#207

s-urbaniak · 2019-04-10T11:32:41Z

/test e2e-aws

brancz · 2019-04-10T12:12:53Z

/lgtm

abhinavdahiya · 2019-04-10T12:32:09Z

/lgtm

wking · 2019-04-10T12:50:29Z

Can we just cut over? I'd have expected keeping both ports open during the transition. I guess we'll see how e2e-aws goes...

brancz · 2019-04-10T12:53:58Z

The insecure port has already been disabled on the components, so I'd be surprised if e2e-aws only turned red now. We (as in monitoring) would have appreciated a transition though instead of a hard break, as this is currently blocking all our PRs as our CI is failing because of these not being available. Now we've almost completed all of the transition though so more for next time 🙂 .

wking · 2019-04-10T13:20:24Z

e2e-aws:

Flaky tests:

[sig-cli] Kubectl client [k8s.io] Simple pod should contain last line of the log [Suite:openshift/conformance/parallel] [Suite:k8s]

Failing tests:

[Feature:Prometheus][Conformance] Prometheus when installed on the cluster should start and expose a secured proxy and unsecured metrics [Suite:openshift/conformance/parallel/minimal]

/retest

wking · 2019-04-10T13:22:17Z

We (as in monitoring) would have appreciated a transition though instead of a hard break, as this is currently blocking all our PRs as our CI is failing because of these not being available.

Sounds like you need to add a stronger monitoring check to the e2e-aws suite, so future changes are gated on not breaking you?

brancz · 2019-04-10T13:23:32Z

Sounds like you need to add a stronger monitoring check to the e2e-aws suite, so future changes are gated on not breaking you?

We had these, but they were disabled for migration purposes 🙃 (without us being informed).

brancz · 2019-04-10T14:36:00Z

The OpenShift API never became available in the test failure, seems unrelated.

/retest

brancz · 2019-04-10T16:14:07Z

That seems like another failure in bringing up the cluster, I'm seeing:

                    {
                        "lastTransitionTime": "2019-04-10T15:19:36Z",
                        "message": "StaticPodsFailing: pods \"kube-apiserver-ip-10-0-130-161.ec2.internal\" not found\nStaticPodsFailing: pods \"kube-apiserver-ip-10-0-168-57.ec2.internal\" not found\nStaticPodsFailing: pods \"kube-apiserver-ip-10-0-148-58.ec2.internal\" not found",
                        "reason": "StaticPodsFailingError",
                        "status": "True",
                        "type": "Failing"
                    },

And

                    {
                        "lastTransitionTime": "2019-04-10T15:19:37Z",
                        "message": "StaticPodsFailing: pods \"kube-controller-manager-ip-10-0-130-161.ec2.internal\" not found\nStaticPodsFailing: pods \"kube-controller-manager-ip-10-0-168-57.ec2.internal\" not found\nStaticPodsFailing: pods \"kube-controller-manager-ip-10-0-148-58.ec2.internal\" not found",
                        "reason": "StaticPodsFailingError",
                        "status": "True",
                        "type": "Failing"
                    },

Among other components. I'm not entirely convinced this is due to this change though as apiserver and openshift-apiserver are failing as well.

Could installer folks have a deeper look? For now restarting, we might just have been unlucky.

/retest

brancz · 2019-04-10T19:10:41Z

Looked into the failure, this time "only" compute nodes and apiserver seem to have failed, but I can't find further logs to find out more.

/retest

openshift-bot · 2019-04-10T20:37:21Z

/retest

Please review the full test history for this PR and help us cut down flakes.

brancz · 2019-04-11T06:30:23Z

failed to create listener: failed to listen on 0.0.0.0:10257: listen tcp 0.0.0.0:10257: bind: address already in use

Seems like a flake as this PRs isn't the one switching this behavior.

/retest

s-urbaniak · 2019-04-11T07:50:18Z

/retest

s-urbaniak · 2019-04-11T10:43:29Z

/retest

s-urbaniak · 2019-04-12T11:43:04Z

At last verified locally this branch successfully and confirmed the control plane is reachable on 10257/10259.

The flakes are addressed in openshift/origin#22543.

s-urbaniak · 2019-04-12T12:26:01Z

/retest

flake PR merge, let's see how this goes

s-urbaniak · 2019-04-12T15:02:36Z

/test e2e-aws

s-urbaniak · 2019-04-12T17:57:20Z

/test e2e-aws

paulfantom · 2019-04-16T18:10:47Z

/retest

s-urbaniak · 2019-04-16T20:22:51Z

/test e2e-aws

s-urbaniak · 2019-04-16T20:25:20Z

@abhinavdahiya this is a chicken/egg problem, as prometheus still tries to scrape insecure ports, this is resolved in openshift/cluster-monitoring-operator#316, but the latter cannot land without this. We temporarily disabled this very e2e test and will reenable once this and the referenced PR land.

s-urbaniak · 2019-04-16T20:39:23Z

side note: openshift/origin#22574 got merged just 6 hours ago due to flakes, so let's observe the current builds.

abhinavdahiya · 2019-04-16T22:08:57Z

e2e-aws is still failing:

[Feature:Prometheus][Conformance] Prometheus when installed on the cluster should start and expose a secured proxy and unsecured metrics [Suite:openshift/conformance/parallel/minimal]

abhinavdahiya · 2019-04-16T23:19:32Z

/test e2e-metal

s-urbaniak · 2019-04-17T05:00:33Z

hmm .. it's still e2e'ing the scheduler metrics 🤔

Apr 16 21:35:17.877: INFO: missing some targets: [no match for map[job:scheduler] with health up and scrape URL ^(http|https)://.*/metrics$]

@paulfantom do you mind to have a look why this is the case?

s-urbaniak · 2019-04-17T05:00:49Z

/retest

s-urbaniak · 2019-04-17T07:37:30Z

/retest

s-urbaniak · 2019-04-17T09:57:10Z

@abhinavdahiya PTAL, aws is green again, I have not enough context about the baremetal failure but it seems unrelated.

abhinavdahiya · 2019-04-17T13:28:32Z

/retest

abhinavdahiya · 2019-04-17T13:29:03Z

/test e2e-aws

brancz · 2019-04-18T05:51:21Z

/retest

s-urbaniak · 2019-04-18T05:59:13Z

fwiw openshift/cluster-monitoring-operator#316 just got merged, so once this lands, we can reenable the metrics e2e tests for the control plane.

brancz · 2019-04-18T09:32:10Z

/retest

brancz · 2019-04-18T12:27:35Z

/retest

brancz · 2019-04-18T13:44:53Z

/retest

abhinavdahiya · 2019-04-18T13:51:51Z

/lgtm

e2e-metal is optional

openshift-ci-robot · 2019-04-18T13:52:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, brancz, s-urbaniak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

brancz · 2019-04-18T14:54:14Z

/retest

brancz · 2019-04-18T15:11:15Z

/retest

trown · 2019-04-18T17:42:28Z

/test e2e-openstack

openshift-bot · 2019-04-18T17:45:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-04-18T17:55:16Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2019-04-18T20:19:12Z

@s-urbaniak: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-metal	`36bb700`	link	`/test e2e-metal`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot requested review from brancz and sttts April 10, 2019 09:54

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 10, 2019

s-urbaniak mentioned this pull request Apr 10, 2019

serviceMonitorKubeScheduler/serviceMonitorKubeControllerManager: enable TLS openshift/cluster-monitoring-operator#316

Merged

controller-manager/scheduler: switch to secure ports

36bb700

Recently, the control plane switched to secure ports in [1] and [2]. This aligns them in the installer. [1] openshift/cluster-kube-scheduler-operator#88 [2] openshift/cluster-kube-controller-manager-operator#207

s-urbaniak force-pushed the secure-ports branch from b166760 to 36bb700 Compare April 10, 2019 09:58

openshift-ci-robot assigned brancz Apr 10, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 10, 2019

openshift-ci-robot assigned abhinavdahiya Apr 10, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 10, 2019

wking mentioned this pull request Apr 10, 2019

Wait for port to be available openshift/cluster-kube-scheduler-operator#90

Merged

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 18, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 18, 2019

openshift-merge-robot merged commit 6e8d748 into openshift:master Apr 18, 2019

maci0 added a commit to maci0/openshift4-ansible that referenced this pull request Apr 18, 2019

align with openshift/installer#1576

f2e1c87

s-urbaniak deleted the secure-ports branch April 23, 2019 07:32

Conversation

s-urbaniak commented Apr 10, 2019

Uh oh!

s-urbaniak commented Apr 10, 2019

Uh oh!

brancz commented Apr 10, 2019

Uh oh!

abhinavdahiya commented Apr 10, 2019

Uh oh!

wking commented Apr 10, 2019

Uh oh!

brancz commented Apr 10, 2019

Uh oh!

wking commented Apr 10, 2019

Uh oh!

wking commented Apr 10, 2019

Uh oh!

brancz commented Apr 10, 2019

Uh oh!

brancz commented Apr 10, 2019

Uh oh!

brancz commented Apr 10, 2019

Uh oh!

brancz commented Apr 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Apr 10, 2019

Uh oh!

brancz commented Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-urbaniak commented Apr 11, 2019

Uh oh!

s-urbaniak commented Apr 11, 2019

Uh oh!

s-urbaniak commented Apr 12, 2019

Uh oh!

s-urbaniak commented Apr 12, 2019

Uh oh!

s-urbaniak commented Apr 12, 2019

Uh oh!

s-urbaniak commented Apr 12, 2019

Uh oh!

paulfantom commented Apr 16, 2019

Uh oh!

s-urbaniak commented Apr 16, 2019

Uh oh!

s-urbaniak commented Apr 16, 2019

Uh oh!

s-urbaniak commented Apr 16, 2019

Uh oh!

abhinavdahiya commented Apr 16, 2019

Uh oh!

abhinavdahiya commented Apr 16, 2019

Uh oh!

s-urbaniak commented Apr 17, 2019

Uh oh!

s-urbaniak commented Apr 17, 2019

Uh oh!

s-urbaniak commented Apr 17, 2019

Uh oh!

s-urbaniak commented Apr 17, 2019

Uh oh!

abhinavdahiya commented Apr 17, 2019

Uh oh!

abhinavdahiya commented Apr 17, 2019

Uh oh!

brancz commented Apr 18, 2019

Uh oh!

s-urbaniak commented Apr 18, 2019

Uh oh!

brancz commented Apr 18, 2019

Uh oh!

brancz commented Apr 18, 2019

Uh oh!

brancz commented Apr 18, 2019

Uh oh!

abhinavdahiya commented Apr 18, 2019

Uh oh!

openshift-ci-robot commented Apr 18, 2019

brancz commented Apr 10, 2019 •

edited

Loading

brancz commented Apr 11, 2019 •

edited

Loading