Disable Kubelet unauthenticated read-only port

[elemental-lf]

Bug

Not a bug per se but a leakage of possibly sensitive information

Environment

All environments AFAIK.

Problem

While researching another issue I just came across the fact that kubelet opens up an unauthenticated HTTP endpoint on port 10255 which can be used to get a dump of all pod manifests (/pods). I just checked on my Typhoon based cluster and the port is indeed open and reachable from the outside. Was this a conscious decision or is this port even used by something at the moment? If not, I'd like to suggest calling kubelet with --read-only-port 0 to disable this unauthenticated access to possibly sensitive information. I've disabled this port for one kubelet of my cluster and haven't seen any ill side effects, yet.
Some context:

• Secure Kubelet's componentconfig defaults while maintaining CLI compatibility kubernetes/kubernetes#59666 (comment)
• Disable the kubelet readonly port kubernetes/kubeadm#732

Desired Behavior

If nothing precludes this I think this port should be closed.

Steps to Reproduce

curl http://$K8S_HOST:10255/pods

[dghubble]

First, the read-only port is not public in the Azure, AWS, GCP, or DigitalOcean firewall configurations. Its limited to traffic between nodes within each cluster.

Removing the need for the kubelet read-only port has been a goal to work toward for a while now. Historically, the kubelet read-only port served as a health checking mechanism for load balancers (since the secure port couldn't be used for this) across various distros for as far back as I can recall (dating back to at least Kubernetes v1.2 or earlier). So yes, its been known for years now.

With recent improvements acorss clouds, load balancers do TCP health checks of apiservers, instead of requiring HTTP health checks (em Google). So that challenge has been addressed in recent releases:

• AWS NLB with an apiserver TCP health check (since v1.10.5) Combine AWS NLBs to use one NLB per cluster #249
• Azure LB with an apiserver TCP probe (since Azure introduced in v1.11.3)
• DigitalOcean doesn't come with a load balancer
• GCP TCP proxy with an apiserver TCP health check (since v1.10.2) - Switch GCP network lb to global TCP proxy lb #190

I've tested disabling the port at various points. Most recently in a branch in May 2018 with v1.10.2. I recall it caused too many breakages to other components at the time (don't have better notes :/). Components still assuming read-only was available. I can revisit it again to see if its safe for inclusion in an upcoming release.

[dghubble]

Correction: Typhoon AWS began using an apiserver TCP health check by v1.9.4.

So Typhoon v1.10.2 (April 2018) achieved apiserver load balancing without requiring the read-only port. But the read-only port was still used for Prometheus to scrape Kubelet and cAdvisor metrics and for Heapster to collect metrics (i.e. my attempts in May).

Typhoon v1.10.3 added Kubelet bearer token authentiation and switched Prometheus to scrape Kubelet and cAdvisor via the Kubelet authenticated API.

Typhoon v1.12.1 still requires the read-only port for Heapster addon (or metrics-server). You'll notice kubectl top commands failing if its removed (or if you remove the firewall rules allowing the traffic). But today, Heapster has options to use the kubelets' secure port. While the Kubernetes default system:heapster ClusterRole lacks the correct GET access to node/stats (see kubernetes-retired/heapster#1936), that's simple enough to patch in the Heapster addon.

[dghubble]

In the next release, the kubelet read-only port is no longer needed.

[elemental-lf]

@dghubble Obviously I was missing a lot of background on this issue. Thank you for the detailed explanation and the historic overview. It is appreciated.