pkg/checkpoint: Try kubelet secureClient, fallback to read-only #1027

dghubble · 2018-11-12T20:37:40Z

Change pod-checkpointer to first try to list pods via the Kubeletsecure (10250) endpoint, then fallback to the Kubelet read-only (10255) endpoint, before defaulting to an empty list of parent pods

Related: #1025

coreosbot · 2018-11-12T20:37:42Z

Can one of the admins verify this patch?

coreosbot · 2018-11-12T20:37:42Z

Can one of the admins verify this patch?

coreosbot · 2018-11-12T20:37:42Z

Can one of the admins verify this patch?

dghubble · 2018-11-12T20:40:02Z

/assign @rphillips

rphillips · 2018-11-12T21:24:40Z

Is there a default timeout on the secure kubeclient?

dghubble · 2018-11-12T22:09:29Z

I don't see one set in client-go rest by default. I could add one for say 15 seconds? More/less? k.secureClient.Get().AbsPath("/pods/").Timeout(15*time.Second).Do().Into(podList)

* Change pod-checkpointer to first try to list pods via the Kubelet secure (10250) endpoint, then fallback to the Kubelet read-only (10255) endpoint, before defaulting to an empty list of parent pods

rphillips · 2018-11-15T19:42:33Z

/ok-to-test

* kubernetes-retired/bootkube#1027

dghubble · 2018-11-26T07:36:07Z

I've been testing with an image quay.io/dghubble/pod-checkpointer:55f0b95cf1471d8179b083c031f43a3a3bcabd0a with this.

On a typical bootkube cluster today (allows kubelet read-only, no ClusterRoleBinding for pod-checkpointer), the checkpointer falls back to the insecure client as usual.

E1126 07:21:31.817761       1 kubelet.go:55] failed to secure list local parent pods, fallback to insecure: Forbidden (user=system:serviceaccount:kube-system:pod-checkpointer, verb=get, resource=nodes, subresource=proxy)

On a cluster with --read-only-port=0, pod-checkpointer previously failed:

# previous 
failed to list local parent pods, assuming none are running: Get http://127.0.0.1:10255/pods/: dial tcp 127.0.0.1:10255: connect: connection refused

With this pod-checkpointer change:

# with ClusterRoleBinding, no error logs
NA
# without ClusterRolebinding, fallback to insecure, insecure is disabled
E1126 07:21:31.817761       1 kubelet.go:55] failed to secure list local parent pods, fallback to insecure: Forbidden (user=system:serviceaccount:kube-system:pod-checkpointer, verb=get, resource=nodes, subresource=proxy)
E1126 07:21:31.818357       1 kubelet.go:58] failed to insecure list local parent pods, assuming none are running: Get http://127.0.0.1:10255/pods/?timeout=15s: dial tcp 127.0.0.1:10255: connect: connection refused

I think this shows the change doesn't regess any current behavior, but adds a way to support disabling the Kubelet read-only port. I can have #1025 be a followup when a pod-checkpointer image is ready.

* kubernetes-retired/bootkube#1027

rphillips · 2018-11-26T14:32:26Z

coreosbot run e2e checkpointer

rphillips · 2018-11-26T15:42:31Z

/approve

k8s-ci-robot · 2018-11-26T15:42:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rphillips]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rphillips · 2018-11-26T15:44:44Z

/lgtm

rphillips · 2018-11-26T15:54:04Z

Pushed the new checkpoint image to quay (tag: 83e25e5968391b9eb342042c435d1b3eeddb2be1)

* Updates pod-checkpointer to prefer the Kubelet secure API (before falling back to the Kubelet read-only API that is disabled on Typhoon clusters since poseidon/typhoon#324) * Previously, pod-checkpointer checkpointed an initial set of pods during bootstrapping so recovery from power cycling clusters was unaffected, but logs were noisy * kubernetes-retired/bootkube#1027 * kubernetes-retired/bootkube#1025

* Updates pod-checkpointer to prefer the Kubelet secure API (before falling back to the Kubelet read-only API that is disabled on Typhoon clusters since #324) * Previously, pod-checkpointer checkpointed an initial set of pods during bootstrapping so recovery from power cycling clusters was unaffected, but logs were noisy * kubernetes-retired/bootkube#1027 * kubernetes-retired/bootkube#1025

* Updates pod-checkpointer to prefer the Kubelet secure API (before falling back to the Kubelet read-only API that is disabled on Typhoon clusters since poseidon/typhoon#324) * Previously, pod-checkpointer checkpointed an initial set of pods during bootstrapping so recovery from power cycling clusters was unaffected, but logs were noisy * kubernetes-retired/bootkube#1027 * kubernetes-retired/bootkube#1025

* Updates pod-checkpointer to prefer the Kubelet secure API (before falling back to the Kubelet read-only API that is disabled on Typhoon clusters since poseidon#324) * Previously, pod-checkpointer checkpointed an initial set of pods during bootstrapping so recovery from power cycling clusters was unaffected, but logs were noisy * kubernetes-retired/bootkube#1027 * kubernetes-retired/bootkube#1025

* Updates pod-checkpointer to prefer the Kubelet secure API (before falling back to the Kubelet read-only API that is disabled on Typhoon clusters since poseidon/typhoon#324) * Previously, pod-checkpointer checkpointed an initial set of pods during bootstrapping so recovery from power cycling clusters was unaffected, but logs were noisy * kubernetes-retired/bootkube#1027 * kubernetes-retired/bootkube#1025

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 12, 2018

k8s-ci-robot requested review from aaronlevy and rphillips November 12, 2018 20:37

dghubble changed the title ~~pkg/checkpoint: Try kubelet secureClient, fallback to insecureClient~~ pkg/checkpoint: Try kubelet secureClient, fallback to read-only Nov 12, 2018

k8s-ci-robot assigned rphillips Nov 12, 2018

pkg/checkpoint: Try Kubelet secureClient, fallback to read-only

55f0b95

* Change pod-checkpointer to first try to list pods via the Kubelet secure (10250) endpoint, then fallback to the Kubelet read-only (10255) endpoint, before defaulting to an empty list of parent pods

dghubble added a commit to poseidon/terraform-render-bootstrap that referenced this pull request Nov 26, 2018

Verify pod-checkpointer image patch

fea0972

* kubernetes-retired/bootkube#1027

dghubble added a commit to poseidon/terraform-render-bootstrap that referenced this pull request Nov 26, 2018

Verify pod-checkpointer image patch

3491284

* kubernetes-retired/bootkube#1027

dghubble added a commit to poseidon/typhoon that referenced this pull request Nov 26, 2018

Verify pod-checkpointer image patch

c3aec47

* kubernetes-retired/bootkube#1027

dghubble added a commit to poseidon/typhoon that referenced this pull request Nov 26, 2018

Verify pod-checkpointer image patch

25a06ce

* kubernetes-retired/bootkube#1027

dghubble mentioned this pull request Nov 26, 2018

Disable Kubelet read-only port 10255 poseidon/typhoon#324

Merged

dghubble added a commit to poseidon/terraform-render-bootstrap that referenced this pull request Nov 26, 2018

Verify pod-checkpointer image patch

94b1bd8

* kubernetes-retired/bootkube#1027

dghubble added a commit to poseidon/typhoon that referenced this pull request Nov 26, 2018

Verify pod-checkpointer image patch

7c3656b

* kubernetes-retired/bootkube#1027

dghubble mentioned this pull request Nov 26, 2018

Disable Kubelet read-only port 10255 #1025

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 26, 2018

k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 26, 2018

k8s-ci-robot merged commit 83e25e5 into kubernetes-retired:master Nov 26, 2018

dghubble mentioned this pull request Nov 26, 2018

Update pod-checkpointer image to try secureClient first #1031

Merged

dghubble deleted the checkpointer-try-secure branch November 26, 2018 17:52

dghubble mentioned this pull request Nov 27, 2018

Update pod-checkpointer image to query Kubelet secure api poseidon/terraform-render-bootstrap#91

Merged

dghubble mentioned this pull request Nov 27, 2018

Update pod-checkpointer image to query Kubelet secure API poseidon/typhoon#346

Merged

dghubble mentioned this pull request Dec 18, 2018

Minor docs fix to re-run master tests #1034

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/checkpoint: Try kubelet secureClient, fallback to read-only #1027

pkg/checkpoint: Try kubelet secureClient, fallback to read-only #1027

dghubble commented Nov 12, 2018 •

edited

Loading

coreosbot commented Nov 12, 2018

coreosbot commented Nov 12, 2018

coreosbot commented Nov 12, 2018

dghubble commented Nov 12, 2018

rphillips commented Nov 12, 2018

dghubble commented Nov 12, 2018 •

edited

Loading

rphillips commented Nov 15, 2018

dghubble commented Nov 26, 2018 •

edited

Loading

rphillips commented Nov 26, 2018

rphillips commented Nov 26, 2018

k8s-ci-robot commented Nov 26, 2018

rphillips commented Nov 26, 2018

rphillips commented Nov 26, 2018 •

edited

Loading

pkg/checkpoint: Try kubelet secureClient, fallback to read-only #1027

pkg/checkpoint: Try kubelet secureClient, fallback to read-only #1027

Conversation

dghubble commented Nov 12, 2018 • edited Loading

coreosbot commented Nov 12, 2018

coreosbot commented Nov 12, 2018

coreosbot commented Nov 12, 2018

dghubble commented Nov 12, 2018

rphillips commented Nov 12, 2018

dghubble commented Nov 12, 2018 • edited Loading

rphillips commented Nov 15, 2018

dghubble commented Nov 26, 2018 • edited Loading

rphillips commented Nov 26, 2018

rphillips commented Nov 26, 2018

k8s-ci-robot commented Nov 26, 2018

rphillips commented Nov 26, 2018

rphillips commented Nov 26, 2018 • edited Loading

dghubble commented Nov 12, 2018 •

edited

Loading

dghubble commented Nov 12, 2018 •

edited

Loading

dghubble commented Nov 26, 2018 •

edited

Loading

rphillips commented Nov 26, 2018 •

edited

Loading