"kubeadm init" doesn't recognize external etcd configuration

[seh]

What keywords did you search in kubeadm issues before filing this one?

• etcd
• external
• generate

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version:

kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version:
  Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  AWS EC2 "t2.medium" instance

• OS (e.g. from /etc/os-release):

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1688.5.3
VERSION_ID=1688.5.3
BUILD_ID=2018-04-03-0547
PRETTY_NAME="Container Linux by CoreOS 1688.5.3 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

• Kernel (e.g. uname -a):
  Linux ip-v-x-y-z 4.14.32-coreos #1 SMP Tue Apr 3 05:21:26 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux

What happened?

I specified the following four values in my kubeadm configuration file:

etcd:
  endpoints: ["https://etcd0.kubernetes.local:2379","https://etcd1.kubernetes.local:2379","https://etcd2.kubernetes.local:2379"]
  caFile: /etc/kubernetes/pki/etcd-ca.crt
  certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
  keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

There are three etcd servers running that allow inbound interaction from the master machine using those TLS assets.

I ran sudo kubeadm init --config , and it proceeds to create all the etcd-related TSL assets on its own:

[...]
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [ip-10-103-0-145] and IPs [10.103.0.145]
[certificates] Generated etcd/healthcheck-client certificate and key.
certificate apiserver-etcd-client is not signed by corresponding CA

I tried moving my own etcd-related assets to the paths expected by kubeadm init, which in this case meant moving file /etcd/kubernetes/pki/etcd-ca.crt to /etcd/kubernetes/pki/etcd/ca.crt. After that and editing the configuration file to match, running kubeadm init again yields the following:

[...]
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
failure loading etcd/ca certificate: couldn't load the private key file /etc/kubernetes/pki/etcd/ca.key: open /etc/kubernetes/pki/etcd/ca.key: no such file or directory

Now kubeadm init wants my etcd CA key, but that shouldn't be required on this machine. This machine is a client to etcd, and should not have the CA key available.

Begrudgingly, just to see what kubeadm would do, I placed the CA key in file /etc/kubernetes/pki/etcd/ca.key, and ran kubeadm init again.

[...]
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [ip-10-103-0-145] and IPs [10.103.0.145]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[...]

That's a little better, but kubedam is still creating the missing (and irrelevant) etcd-related TLS assets.

What you expected to happen?

kubeadm init should find the four configuration values set in the configuration file (etcd.endpoints, etcd.caFile, etcd.certFile, and etcd.keyFile) and proceed assuming that it doesn't need to create anything to make use of etcd. It should try to use my existing etcd cluster, and only if it finds that it can't use it should it complain further.

How to reproduce it (as minimally and precisely as possible)?

• Create a separate etcd cluster using a separate CA.
• Place the etcd CA certificate on a master machine at, say, /etc/kubernetes/pki/etcd-ca.crt.
• Create client key and certificate signed by the etcd CA for use from the master machine.
• Place these files on a master machine at /etc/kubernetes/pki/apiserver-etcd-client.{crt,key}.
• Create a kubeadm configuration file specifying these server addresses and files:

etcd:
  endpoints: ["https://etcd0.kubernetes.local:2379","https://etcd1.kubernetes.local:2379","https://etcd2.kubernetes.local:2379"]
  caFile: /etc/kubernetes/pki/etcd-ca.crt
  certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
  keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

• Run kubeadm init --config and observe its actions and complaints.

Anything else we need to know?

I couldn't find any documented flags or configuration file fields that I could specify to force kubeadm's hand into accepting that I have an existing etcd cluster.

What am I missing here?

[seh]

Please also see this long discussion in the "kubeadm" Slack channel with @mauilion.

[seh]

If I move my etcd CA certificate to /etc/kubernetes/pki/etcd/ca.crt and put my etcd CA key into place there as /etc/kubernetes/pki/etcd/ca.key, then kubeadm init makes it further to complaining about the missing front proxy CA key, which shouldn't be necessary either, per #752. However, if I put that CA key into place as /etc/kubernetes/pki/front-proxy-ca.key, then kubeadm init completes.

That's two separate problems; #752 is fixed by kubernetes/kubernetes#62643, but I assume that I don't have that fix incorporated in my version of kubeadm yet. (I'm using version 1.10.2.)

During my discussion with @mauilion last night, we noticed that the "certificates" phase (as in kubeadm alpha phase certs) has two problems:

• It proceeds without regard for there being an external etcd cluster.
• It demands that the etcd CA key be present, even though it doesn't actually validate it.
  In kubeadm/app/phases/certs/certs.go, the CreateEtcdCACertAndKeyFiles function calls on writeCertificateAuthorithyFilesIfNotExist, which calls pkiutil.TryLoadCertAndKeyFromDisk. Note that even though this caller throws away the returned CA key, pkiutil.TryLoadCertAndKeyFromDisk demands that both files exist. Instead, I think we should use pkiutil.TryLoadCertFromDisk here, which tolerates the lack of the key.

More broadly, kubeadm init skips the entire "certificates" phase if the Kubernetes CA key is absent, but I think this is too coarse of a decision. In my case, I've generated all the certificates myself, but I don't want to take responsibility for generating the KUBECONFIG files, and I'd like to be able to use the csrsigner controller, so I put the Kubernetes CA key in place.

Could we allow for the Kubernetes CA key to be present, but allow for the etcd CA key to be absent? That is, we'd have the notion of "external etcd CA mode," where none of the etcd TLS assets need to be generated by kubeadm.

[seh]

I see that in the "Creating HA clusters with kubeadm" document, there's a section "Acquire etcd certs" that reads:

2. Run the following:
   mkdir -p /etc/kubernetes/pki/etcd
   scp root@:/etc/kubernetes/pki/etcd/ca.pem /etc/kubernetes/pki/etcd
   scp root@:/etc/kubernetes/pki/etcd/client.pem /etc/kubernetes/pki/etcd
   scp root@:/etc/kubernetes/pki/etcd/client-key.pem /etc/kubernetes/pki/etcd

That would copy only three files into place: the etcd CA certificate, the etcd client certificate, and the etcd client private key. The instructions that follow on invoking kubeadm init don't say anything about needing the etcd CA private key to be present.

[mauilion]

In my testing I was able to copy the kubernetes ca.crt and key as well as the etcd CA and client certs in place. Kubeadm would then generate the correct certs and use the existing kubernetes ca and key and etcd certs. I'm still unclear on the difference in our setups. I see a few confusing points above.

The certificates that kubeadm looks for when determining whether to generate etcd certs are identified in you kubeadm config.

Placing your etcd CA cert at /etc/kubernetes/pki/ca.crt will likely mean that kubeadm will use that as the kubernetes ca.

[seh]

Please note that my etcd CA certificate is in /etc/kubernetes/pki/etcd-ca.crt, not ca.crt. The latter is there too, and is my Kubernetes CA certificate.

(I updated my earlier comment to clarify that point. You're right that I had written the wrong path there.)