kubeadm: "failure loading ca certificate: the certificate is not valid yet" #2749

kierenj · 2018-04-19T08:03:21Z

Please provide the following details:

Environment:

Minikube version (use minikube version): 0.26.1

OS (e.g. from /etc/os-release): Windows 10 (build 16299)
VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperv
ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): file://C:/Users/kiere/.minikube/cache/iso/minikube-v0.26.0.iso
Install tools: ?
Others: ?

What happened:

C:\Users\kiere> sudo minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Finished Downloading kubelet v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0419 08:54:23.948090    2468 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data
--ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
                                                                                                                                                                [WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
failure loading ca certificate: the certificate is not valid yet
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1

What you expected to happen:
Minikube VM to start successfully.

How to reproduce it (as minimally and precisely as possible):
Follow the exact steps here: https://medium.com/@JockDaRock/minikube-on-windows-10-with-hyper-v-6ef0f4dc158c

Output of minikube logs (if applicable):

F0419 09:01:40.025689   19032 logs.go:50] Error getting cluster bootstrapper: getting kubeadm bootstrapper: getting ssh client: Error creating new ssh host from driver: Error getting ssh host name for driver: Host is not running

Anything else do we need to know:
Would be great to understand if the "WARNING Swap" is relevant/related. I'm unsure if it's referring to the host OS, or VM OS - or how to remedy?

The text was updated successfully, but these errors were encountered:

fabriziocucci · 2018-04-28T15:45:25Z

Same issue on macOS Sierra.

AmazingTurtle · 2018-05-04T18:21:08Z

Same here. If you delete and recreate the minikube, it's saying that the apiserver certificate is wrong. I'm using VirtualBox. The vm driver is not the cause.

I can confirm that this issue does not occur in v0.25.2, starts perfectly.

$ minikube delete && minikube start
Deleting local Kubernetes cluster...
Machine deleted.
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0504 20:16:11.725862   14520 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[certificates] Using the existing ca certificate and key.
        [WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
failure loading apiserver certificate: the certificate is not valid yet
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
        minikube config set WantReportErrorPrompt false
================================================================================

chrobotm · 2018-05-13T17:48:50Z

same issue here

fore5fire · 2018-05-15T04:19:59Z

Experiencing the same issue, Linux with kvm2 driver, minikube v0.26.1. I started getting this error when trying to start my previously-working cluster:

minikube start --vm-driver kvm2
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0514 21:16:43.158255    5693 start.go:281] Error restarting cluster:  running cmd: 
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

After trying to delete and recreate the cluster, I get the same issue as OP. After getting the "the certificate is not yet valid" error, If I try to start the vm again without deleting the vm first I get the same error that I was having before I deleted the cluster.

jstangroome · 2018-05-18T04:24:25Z

On Windows 10 version 1803 (OS Build 17134.48) with Virtualbox 5.2.12 (also 5.2.6) and Minikube 0.27.0 (also 0.26.1) I get the "certificate is not yet valid" error during a minikube start with no existing Minikube VM.

Comparing the host time to the VM time, I see a 3 second difference:

PS > get-date ;  minikube ssh -- date --utc ; get-date
Friday, 18 May 2018 14:21:47
Fri May 18 04:21:44 UTC 2018
Friday, 18 May 2018 14:21:47

Comparing the cert validity time to the cert file modified time I see the same 3 second difference:

$ minikube ssh -- stat /var/lib/localkube/certs/apiserver.crt
  File: /var/lib/localkube/certs/apiserver.crt
  Size: 1298            Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 3670105     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-05-18 04:41:51.768791568 +0000
Modify: 2018-05-18 04:41:50.237026124 +0000
Change: 2018-05-18 04:41:50.237026124 +0000
 Birth: -

$ minikube ssh -- cat /var/lib/localkube/certs/apiserver.crt | openssl x509 -noout -text | grep alidity -A2
        Validity
            Not Before: May 18 04:41:53 2018 GMT
            Not After : May 18 04:41:53 2019 GMT

Update: after leaving the host running for a few days without reboots, the clock discrepancy between the host and newly created VMs has reduced to subsecond and the "certificate is not yet valid" error is no longer encountered.

Since the clock discrepancy is between the host and a new VM (as opposed to existing VMs) and since it has impacted users with Virtualbox and users with Hyper-V, implementing a change to be less sensitive to time variations during minikube start would seem prudent.

jstangroome · 2018-05-18T06:09:29Z

If I run minikube start again following the "certificate is not yet valid" error but after enough time has elapsed for the VM clock to be after the certificate's Not Before time, then I get the same error as reported above by @LSmith130:

Starting cluster components...
E0518 15:54:06.919233   13964 start.go:281] Error restarting cluster:  running cmd:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

Note the above output is with minikube log level 99.

If I run the supposedly failing command directly with minikube ssh it does not fail:

PS > minikube ssh -- "sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
>> sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
>> sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
>> sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml ; echo EXITED `$?"
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [minikube] and IPs [192.168.99.101]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/var/lib/localkube/certs/"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
EXITED 0

Trying minikube start again after verifying the command succeeds once again reports failure for the same command.

However, minikube status reports healthy:

PS > minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.101

Meanwhile the storage-provisioner and kubernetes-dashboard are both in a crash loop:

PS > kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
kube-system   etcd-minikube                           1/1       Running            0          7m
kube-system   kube-addon-manager-minikube             1/1       Running            0          6m
kube-system   kube-apiserver-minikube                 1/1       Running            0          7m
kube-system   kube-controller-manager-minikube        1/1       Running            0          7m
kube-system   kube-scheduler-minikube                 1/1       Running            0          7m
kube-system   kubernetes-dashboard-5498ccf677-t9jff   0/1       CrashLoopBackOff   5          7m
kube-system   storage-provisioner                     0/1       CrashLoopBackOff   5          7m

PS > kubectl logs -n kube-system storage-provisioner
F0518 06:04:00.140866       1 main.go:37] Error getting server version: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout

PS > kubectl logs -n kube-system kubernetes-dashboard-5498ccf677-t9jff
2018/05/18 06:06:47 Starting overwatch
2018/05/18 06:06:47 Using in-cluster config to connect to apiserver
2018/05/18 06:06:47 Using service account token for csrf signing
2018/05/18 06:06:47 No request provided. Skipping authorization
2018/05/18 06:07:17 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

For reference, the kubeadm.yaml file contents:

PS > minikube ssh -- sudo cat /var/lib/kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
  advertiseAddress: 192.168.99.101
  bindPort: 8443
kubernetesVersion: v1.10.0
certificatesDir: /var/lib/localkube/certs/
networking:
  serviceSubnet: 10.96.0.0/12
etcd:
  dataDir: /data
nodeName: minikube
apiServerExtraArgs:
  admission-control: "Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"

chrobotm · 2018-05-20T17:18:39Z

v0.27 is working for me on mac osx

aseempatni · 2018-05-21T07:34:45Z

I'm getting failure loading ca certificate: the certificate is not valid yeton Mac with minikube version: v0.27.0

kierenj · 2018-05-21T07:36:02Z

@mbotmcc @aseempatni - see the linked pull request above, it looks like a fix is identified and ready to merge - but isn't merged yet. I don't think 0.27 would have anything other than intermittent success

chrobotm · 2018-05-21T07:45:01Z

@aseempatni I got this error too but after I deleted everything in .minikube and downloaded everything again it worked

addisonhuddy · 2018-06-09T20:46:13Z

Getting the same issue.

E0609 14:36:04.659245   61628 start.go:276] Error starting cluster:  ...

Workaround: minikube ssh and run the init command

sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

kubectl can now connect to minikube. Still have no idea what is actually going on.

h0nor · 2018-06-12T18:38:30Z

I first installed v0.27.0 on MacOS 10.11.6 El Capitan.
Tried many things but it didn't work.

To make it work this is what I did:

cd /Users/user_name/.minikube/cache
rm -Rf localkube
If you have another folder like "v1.10.0" or "v1.9.0"
rm -Rf v1.10.0
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
minikube start

Thanks @AmazingTurtle for referring to a version that actually works!

Arnavion · 2018-06-14T00:50:06Z

The PR with the fix should be in 0.28.0, so this should be fixed now.

fejta-bot · 2018-09-12T00:52:41Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

tstromberg · 2018-09-19T17:40:51Z

Fixed in 0.28.0.

tstromberg · 2018-11-30T17:39:10Z

Reopening because we're seeing this issue in our CI environment, although it's only showing up in VirtualBox at the moment. The root cause is effectively that time in the VM is lagging compared to the host.

example:

[certificates] Generated etcd/healthcheck-client certificate and key.                                                               
failure loading etcd/ca certificate authority: the certificate is not valid yet

tstromberg · 2018-12-14T21:32:06Z

Doing some reading, vbox doesn't sync time on VM start:

https://vilimpoc.org/blog/2014/02/09/better-time-synchronization-with-virtualbox-guest-additions/

I'm working on a PR to configure this.

ivan-section-io mentioned this issue May 19, 2018

Give certificates some NotBefore leeway for VMs clocks that lag host clocks #2823

Merged

Arnavion mentioned this issue Jun 9, 2018

Travis CI Arnavion/k8s-openapi#5

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 12, 2018

tstromberg closed this as completed Sep 19, 2018

tstromberg added tls ev/certificate-errors failed due to certificate issues labels Sep 19, 2018

tstromberg changed the title ~~Hyper-V minikube start error: "failure loading ca certificate: the certificate is not valid yet"~~ start error: "failure loading ca certificate: the certificate is not valid yet" Sep 19, 2018

tstromberg reopened this Nov 30, 2018

tstromberg changed the title ~~start error: "failure loading ca certificate: the certificate is not valid yet"~~ kubeadm: "failure loading ca certificate: the certificate is not valid yet" Nov 30, 2018

tstromberg mentioned this issue Dec 11, 2018

Linux VirtualBox TestStartStop: failure loading etcd/ca certificate authority: the certificate is not valid yet #3433

Closed

tstromberg self-assigned this Dec 14, 2018

balopat mentioned this issue Dec 19, 2018

Enable host DNS resolution in virtualbox driver by default #3453

Merged

tstromberg mentioned this issue Dec 20, 2018

VirtualBox time cleanup: sync on boot, don't run timesyncd #3476

Merged

tstromberg closed this as completed in #3476 Dec 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: "failure loading ca certificate: the certificate is not valid yet" #2749

kubeadm: "failure loading ca certificate: the certificate is not valid yet" #2749

kierenj commented Apr 19, 2018

fabriziocucci commented Apr 28, 2018

AmazingTurtle commented May 4, 2018 •

edited

Loading

chrobotm commented May 13, 2018

fore5fire commented May 15, 2018

jstangroome commented May 18, 2018 •

edited

Loading

jstangroome commented May 18, 2018

chrobotm commented May 20, 2018

aseempatni commented May 21, 2018

kierenj commented May 21, 2018

chrobotm commented May 21, 2018

addisonhuddy commented Jun 9, 2018 •

edited

Loading

h0nor commented Jun 12, 2018

Arnavion commented Jun 14, 2018 •

edited

Loading

fejta-bot commented Sep 12, 2018

tstromberg commented Sep 19, 2018

tstromberg commented Nov 30, 2018 •

edited

Loading

tstromberg commented Dec 14, 2018

kubeadm: "failure loading ca certificate: the certificate is not valid yet" #2749

kubeadm: "failure loading ca certificate: the certificate is not valid yet" #2749

Comments

kierenj commented Apr 19, 2018

fabriziocucci commented Apr 28, 2018

AmazingTurtle commented May 4, 2018 • edited Loading

chrobotm commented May 13, 2018

fore5fire commented May 15, 2018

jstangroome commented May 18, 2018 • edited Loading

jstangroome commented May 18, 2018

chrobotm commented May 20, 2018

aseempatni commented May 21, 2018

kierenj commented May 21, 2018

chrobotm commented May 21, 2018

addisonhuddy commented Jun 9, 2018 • edited Loading

h0nor commented Jun 12, 2018

Arnavion commented Jun 14, 2018 • edited Loading

fejta-bot commented Sep 12, 2018

tstromberg commented Sep 19, 2018

tstromberg commented Nov 30, 2018 • edited Loading

tstromberg commented Dec 14, 2018

AmazingTurtle commented May 4, 2018 •

edited

Loading

jstangroome commented May 18, 2018 •

edited

Loading

addisonhuddy commented Jun 9, 2018 •

edited

Loading

Arnavion commented Jun 14, 2018 •

edited

Loading

tstromberg commented Nov 30, 2018 •

edited

Loading