Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.35.0 hangs waiting for DNS pods before CNI is deployed #3852

Closed
anitgandhi opened this issue Mar 11, 2019 · 7 comments · Fixed by #3994
Closed

v0.35.0 hangs waiting for DNS pods before CNI is deployed #3852

anitgandhi opened this issue Mar 11, 2019 · 7 comments · Fixed by #3994
Labels
area/dns DNS issues co/coredns CoreDNS related issues kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Milestone

Comments

@anitgandhi
Copy link

On v0.35.0, I'm seeing an issue where minikube start waits for the core pods to come up, but hangs on dns. Looking at kubectl get pods --all-namespaces , I can see that the 2 coredns pods are pending, similar to #3567 . Of course, this makes sense since there isn't a CNI deployed yet. It also happens even if I don't set --container-runtime and let it fall back to the default of Docker.

This issue didn't seem to present itself on v0.34.1, using the same command. I believe this is because of the behavior introduced in #3774 . On v0.34.1 and prior, minikube doesn't block on waiting for DNS pods to come up; instead, it would continue and eventually clean exit, at which point my immediate next step of a CNI deployment would take over and after than, DNS pods would successfully start.

When I removed the CNI related lines in minikube start, everything worked fine. This makes sense since once I remove those lines, the behavior introduced in #3617 would take effect, enabling the default mybridge CNI. The issue with this is that coredns pods will be managed by that CNI instead of the one I deployed (unless I restart the coredns pods)

Perhaps this is expected behavior, in which case, this can be closed. But in my opinion, when users want to bring their own CNI, there should be an option for that.

I guess the tl;dr is that the change in waiting for DNS pods to come up before a CNI is up (when not using the default CNI) instead of continuing is what's causing this issue for me 😅

Logs

→  make destroy start
minikube delete
🔥  Deleting "minikube" from virtualbox ...
💔  The "minikube" cluster has been deleted.
minikube start \
	--kubernetes-version=v1.13.4 \
	--vm-driver=virtualbox \
	--network-plugin=cni \
	--extra-config=kubelet.network-plugin=cni \
	--container-runtime=containerd
😄  minikube v0.35.0 on linux (amd64)
🔥  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿  Downloading Minikube ISO ...
 184.42 MB / 184.42 MB [============================================] 100.00% 0s
📶  "minikube" IP address is 192.168.99.101
📦  Configuring containerd as the container runtime ...
✨  Preparing Kubernetes environment ...
    ▪ kubelet.network-plugin=cni
🚜  Pulling images required by Kubernetes v1.13.4 ...
🚀  Launching Kubernetes v1.13.4 using kubeadm ... 
⌛  Waiting for pods: apiserver proxy etcd scheduler controller addon-manager dns^CMakefile:31: recipe for target 'start' failed
make: *** [start] Interrupt

OS: Ubuntu 18.04 with VirtualBox 6.0

minikube logs output (and a snippet showing there was no default CNI config deployed):

→  minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ ls /etc/cni/net.d/
$ exit
logout

→  minikube logs
==> kube-apiserver <==
I0311 16:42:06.182456       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-scheduler
I0311 16:42:06.222128       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:aws-cloud-provider
I0311 16:42:06.267371       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node
I0311 16:42:06.307287       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:volume-scheduler
I0311 16:42:06.344247       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:attachdetach-controller
I0311 16:42:06.385029       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
I0311 16:42:06.429862       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:cronjob-controller
I0311 16:42:06.463083       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:daemon-set-controller
I0311 16:42:06.502249       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:deployment-controller
I0311 16:42:06.543191       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:disruption-controller
I0311 16:42:06.588089       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpoint-controller
I0311 16:42:06.628840       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
I0311 16:42:06.665900       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
I0311 16:42:06.703688       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
I0311 16:42:06.742362       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
I0311 16:42:06.781993       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
I0311 16:42:06.823489       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
I0311 16:42:06.866002       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
I0311 16:42:06.905876       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
I0311 16:42:06.947521       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replicaset-controller
I0311 16:42:06.982643       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replication-controller
I0311 16:42:07.027704       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:resourcequota-controller
I0311 16:42:07.067797       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:route-controller
I0311 16:42:07.106888       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-account-controller
I0311 16:42:07.148737       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-controller
I0311 16:42:07.183301       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:statefulset-controller
I0311 16:42:07.226810       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:ttl-controller
I0311 16:42:07.265432       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:certificate-controller
I0311 16:42:07.302978       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
I0311 16:42:07.343175       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pv-protection-controller
I0311 16:42:07.381118       1 controller.go:608] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0311 16:42:07.387241       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader in kube-system
I0311 16:42:07.423549       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0311 16:42:07.462063       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0311 16:42:07.502830       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0311 16:42:07.543002       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0311 16:42:07.583155       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0311 16:42:07.633127       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0311 16:42:07.662089       1 controller.go:608] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0311 16:42:07.671406       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0311 16:42:07.702607       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0311 16:42:07.742236       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0311 16:42:07.781912       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0311 16:42:07.822549       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0311 16:42:07.862084       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0311 16:42:09.266106       1 controller.go:608] quota admission added evaluator for: serviceaccounts
I0311 16:42:09.278468       1 controller.go:608] quota admission added evaluator for: deployments.apps
I0311 16:42:09.317257       1 controller.go:608] quota admission added evaluator for: daemonsets.apps
I0311 16:42:15.979516       1 controller.go:608] quota admission added evaluator for: replicasets.apps
I0311 16:42:16.192935       1 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
==> kube-scheduler <==
E0311 16:42:01.934600       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0311 16:42:01.943534       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0311 16:42:01.948248       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0311 16:42:01.950653       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0311 16:42:01.952305       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0311 16:42:01.953591       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0311 16:42:02.916924       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0311 16:42:02.924914       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0311 16:42:02.927908       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0311 16:42:02.929180       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0311 16:42:02.936238       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0311 16:42:02.945082       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0311 16:42:02.949481       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0311 16:42:02.953093       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0311 16:42:02.954416       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0311 16:42:02.955551       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0311 16:42:03.918003       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0311 16:42:03.928097       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0311 16:42:03.928779       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0311 16:42:03.930455       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0311 16:42:03.937097       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0311 16:42:03.946340       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0311 16:42:03.950591       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0311 16:42:03.955009       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0311 16:42:03.955947       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0311 16:42:03.957533       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0311 16:42:04.919006       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0311 16:42:04.928886       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0311 16:42:04.930463       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0311 16:42:04.932280       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0311 16:42:04.938666       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0311 16:42:04.949179       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0311 16:42:04.951987       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0311 16:42:04.960084       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0311 16:42:04.960120       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0311 16:42:04.960198       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0311 16:42:05.920193       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0311 16:42:05.930625       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0311 16:42:05.931193       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0311 16:42:05.933218       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0311 16:42:05.940181       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0311 16:42:05.953143       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0311 16:42:05.953967       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0311 16:42:05.961464       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0311 16:42:05.962452       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0311 16:42:05.965706       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
I0311 16:42:07.844083       1 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
I0311 16:42:07.944664       1 controller_utils.go:1034] Caches are synced for scheduler controller
I0311 16:42:07.944699       1 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-scheduler...
I0311 16:42:07.949819       1 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler
==> kubelet <==
-- Logs begin at Mon 2019-03-11 16:39:23 UTC, end at Mon 2019-03-11 16:46:07 UTC. --
Mar 11 16:42:16 minikube kubelet[2889]: I0311 16:42:16.296072    2889 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/a0f947dc-441c-11e9-8f44-080027789d9d-kube-proxy") pod "kube-proxy-26n6l" (UID: "a0f947dc-441c-11e9-8f44-080027789d9d")
Mar 11 16:42:16 minikube kubelet[2889]: I0311 16:42:16.296111    2889 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/a0f947dc-441c-11e9-8f44-080027789d9d-lib-modules") pod "kube-proxy-26n6l" (UID: "a0f947dc-441c-11e9-8f44-080027789d9d")
Mar 11 16:42:16 minikube kubelet[2889]: I0311 16:42:16.296144    2889 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/a0f947dc-441c-11e9-8f44-080027789d9d-xtables-lock") pod "kube-proxy-26n6l" (UID: "a0f947dc-441c-11e9-8f44-080027789d9d")
Mar 11 16:42:16 minikube kubelet[2889]: I0311 16:42:16.296158    2889 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-tp8w8" (UniqueName: "kubernetes.io/secret/a0f947dc-441c-11e9-8f44-080027789d9d-kube-proxy-token-tp8w8") pod "kube-proxy-26n6l" (UID: "a0f947dc-441c-11e9-8f44-080027789d9d")
Mar 11 16:42:19 minikube kubelet[2889]: E0311 16:42:19.840469    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:24 minikube kubelet[2889]: E0311 16:42:24.843002    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:29 minikube kubelet[2889]: E0311 16:42:29.845972    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:34 minikube kubelet[2889]: E0311 16:42:34.847998    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:39 minikube kubelet[2889]: E0311 16:42:39.855800    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:44 minikube kubelet[2889]: E0311 16:42:44.857900    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:49 minikube kubelet[2889]: E0311 16:42:49.859048    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:54 minikube kubelet[2889]: E0311 16:42:54.862170    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:42:59 minikube kubelet[2889]: E0311 16:42:59.865427    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:04 minikube kubelet[2889]: E0311 16:43:04.866615    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:09 minikube kubelet[2889]: E0311 16:43:09.868281    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:14 minikube kubelet[2889]: E0311 16:43:14.871803    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:19 minikube kubelet[2889]: E0311 16:43:19.875428    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:24 minikube kubelet[2889]: E0311 16:43:24.877814    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:29 minikube kubelet[2889]: E0311 16:43:29.882764    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:34 minikube kubelet[2889]: E0311 16:43:34.886063    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:39 minikube kubelet[2889]: E0311 16:43:39.888219    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:44 minikube kubelet[2889]: E0311 16:43:44.890568    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:49 minikube kubelet[2889]: E0311 16:43:49.893040    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:54 minikube kubelet[2889]: E0311 16:43:54.895526    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:43:59 minikube kubelet[2889]: E0311 16:43:59.896819    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:04 minikube kubelet[2889]: E0311 16:44:04.898843    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:09 minikube kubelet[2889]: E0311 16:44:09.900766    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:14 minikube kubelet[2889]: E0311 16:44:14.902510    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:19 minikube kubelet[2889]: E0311 16:44:19.905559    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:24 minikube kubelet[2889]: E0311 16:44:24.909550    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:29 minikube kubelet[2889]: E0311 16:44:29.912064    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:34 minikube kubelet[2889]: E0311 16:44:34.915122    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:39 minikube kubelet[2889]: E0311 16:44:39.919051    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:44 minikube kubelet[2889]: E0311 16:44:44.923502    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:49 minikube kubelet[2889]: E0311 16:44:49.925157    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:54 minikube kubelet[2889]: E0311 16:44:54.927327    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:44:59 minikube kubelet[2889]: E0311 16:44:59.930102    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:04 minikube kubelet[2889]: E0311 16:45:04.931993    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:09 minikube kubelet[2889]: E0311 16:45:09.933774    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:14 minikube kubelet[2889]: E0311 16:45:14.936749    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:19 minikube kubelet[2889]: E0311 16:45:19.941550    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:24 minikube kubelet[2889]: E0311 16:45:24.946115    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:29 minikube kubelet[2889]: E0311 16:45:29.948883    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:34 minikube kubelet[2889]: E0311 16:45:34.953450    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:39 minikube kubelet[2889]: E0311 16:45:39.957350    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:44 minikube kubelet[2889]: E0311 16:45:44.959829    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:49 minikube kubelet[2889]: E0311 16:45:49.961782    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:54 minikube kubelet[2889]: E0311 16:45:54.964283    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:45:59 minikube kubelet[2889]: E0311 16:45:59.966674    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Mar 11 16:46:04 minikube kubelet[2889]: E0311 16:46:04.969293    2889 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
@anitgandhi
Copy link
Author

Also, I just realized that doing this

minikube start \
	--kubernetes-version=v1.13.4 \
	--vm-driver=virtualbox \
	--network-plugin=cni \
	--container-runtime=containerd

does NOT set --enable-default-cni . Based on #3567 and #3617 , is it supposed to, when --container-runtime is set?

@tstromberg tstromberg changed the title minikube v0.35.0 start hangs waiting for DNS pods before custom CNI is deployed v0.35.0 start hangs waiting for DNS pods before CNI is deployed Mar 13, 2019
@tstromberg tstromberg changed the title v0.35.0 start hangs waiting for DNS pods before CNI is deployed v0.35.0 hangs waiting for DNS pods before CNI is deployed Mar 13, 2019
@tstromberg
Copy link
Contributor

This sounds like a bug with how we handle CNI. Surprisingly, we have no integration tests for this!

Question: shouldn't CNI come up before DNS? Alternatively, should we just not block on DNS if CNI is provided?

@tstromberg tstromberg added kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. area/dns DNS issues co/coredns CoreDNS related issues labels Mar 13, 2019
@anitgandhi
Copy link
Author

Yeah I think CNI should come up before DNS, hence we should not block if the user chooses to bring their own. In my opinion, this is how it should be:

if --network-plugin=cni is given:
- if --enable-default-cni=true --> deploy the default CNI, and wait for DNS pods
- else, explicitly expect the user to provide their own CNI, and do not block waiting on CNI.

It should be clear in documentation / usage that if ONLY --network-plugin=cni is given, then users are expected to deploy a CNI, and that DNS pods may not work until done so.

It becomes a little fuzzy if an alternate container runtime is chosen. If the user wants a custom CNI, maybe they must set --enable-default-cni=false? And then minikube won't wait block waiting on CNI. Alternatively, we could partially change the default behavior and never implicitly set --enable-default-cni at all, and make it always opt-in?

I'm personally of the opinion that CNI should become the default instead of kubenet and then the above becomes very different, but I realize that's not entirely relevant here 😅

I'm happy to help contribute the above once a consensus is reached 😄

@tstromberg tstromberg added this to the v1.0.0 milestone Mar 25, 2019
@tstromberg
Copy link
Contributor

Note: I believe this was addressed by #3896 - and will be part of the v1.0 release.

@tstromberg
Copy link
Contributor

tstromberg commented Mar 26, 2019

Just to confirm that 1.0 will address it and that it's covered by our integration tests, what command-line should I use to replicate this issue?

@anitgandhi
Copy link
Author

@tstromberg I think on minikube v0.35.0, just this much should replicate it:

minikube start \
	--network-plugin=cni \
	--extra-config=kubelet.network-plugin=cni

(Not exactly sure if that --extra-config is redundant).

The specific command that failed for me on v0.35.0 was this, but based on testing it reproduced with less flags:

minikube start \
	--kubernetes-version=v1.13.4 \
	--vm-driver=virtualbox \
	--network-plugin=cni \
	--extra-config=kubelet.network-plugin=cni \
	--container-runtime=containerd

@tstromberg
Copy link
Contributor

@anitgandhi - thanks! I've sent out a PR to add an example CNI test so that we don't break this feature again in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dns DNS issues co/coredns CoreDNS related issues kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants