Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hyperv restart: waiting for component=kube-apiserver: timed out waiting for the condition #3836

Closed
anvvishnu opened this issue Mar 9, 2019 · 8 comments
Closed

hyperv restart: waiting for component=kube-apiserver: timed out waiting for the condition #3836

anvvishnu opened this issue Mar 9, 2019 · 8 comments
Labels
co/hyperv HyperV related issues ev/apiserver-timeout timeout talking to the apiserver kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it. triage/obsolete Bugs that no longer occur in the latest stable release

Comments

@anvvishnu
Copy link

anvvishnu commented Mar 9, 2019 •
edited by tstromberg
Loading

I have installed minikube for windows and made necessary changes for Hyper V but still I get this error when I try to restart (I did delete and try again as well but no luck)

PS C:\WINDOWS\system32> minikube start --vm-driver=hyperv --hyperv-virtual-switch \
"Primary Virtual Switch" --kubernetes-version="v1.13.0"
o   minikube v0.35.0 on windows (amd64)
x   Kubernetes downgrade is not supported, will continue to use v1.13.4
i   Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
:   Re-using the currently running hyperv VM for "minikube" ...
:   Waiting for SSH access ...
-   "minikube" IP address is 192.168.1.155
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
-   Pulling images required by Kubernetes v1.13.4 ...
:   Relaunching Kubernetes v1.13.4 using kubeadm ...
:   Waiting for pods: apiserver!   Error restarting cluster: wait: waiting for component=kube-apiserver: timed out waiting for the condition
@tstromberg tstromberg changed the title Unable to start minikube on Windows 10 hyperv restart: waiting for component=kube-apiserver: timed out waiting for the condition Mar 9, 2019
@tstromberg
Copy link
Contributor

Thank you for the bug report! I'm sorry to hear that minikube failed in this environment. Do you mind providing the output of:

minikube status

and

minikube logs

It's possible that creating a new cluster or running minikube delete will get passed this error, but I'm very curious what set of messages we can get out of the log so that we can prevent the issue from happening in the future, or at least detect it.

Thank you!

@tstromberg tstromberg added kind/bug Categorizes issue or PR as related to a bug. co/hyperv HyperV related issues triage/needs-information Indicates an issue needs more information in order to work on it. ev/apiserver-timeout timeout talking to the apiserver labels Mar 9, 2019
@anvvishnu
Copy link
Author

anvvishnu commented Mar 9, 2019 •
edited by tstromberg
Loading

I did a verbose while restarting and got the following information

I0309 12:15:05.258184   15284 kubernetes.go:123] error getting Pods with label selector "component=kube-apiserver" [Get https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 192.168.1.154:8443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
I0309 12:15:05.768854   15284 round_trippers.go:383] GET https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver
I0309 12:15:05.768854   15284 round_trippers.go:390] Request Headers:
I0309 12:15:05.771918   15284 round_trippers.go:393]     Accept: application/json, */*
I0309 12:15:05.778858   15284 round_trippers.go:393]     User-Agent: minikube.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0309 12:15:26.804272   15284 round_trippers.go:408] Response Status:  in 21020 milliseconds
I0309 12:15:26.804272   15284 kubernetes.go:123] error getting Pods with label selector "component=kube-apiserver" [Get https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 192.168.1.154:8443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
I0309 12:15:27.276365   15284 round_trippers.go:383] GET https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver
I0309 12:15:27.276365   15284 round_trippers.go:390] Request Headers:
I0309 12:15:27.278377   15284 round_trippers.go:393]     Accept: application/json, */*
I0309 12:15:27.285371   15284 round_trippers.go:393]     User-Agent: minikube.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0309 12:15:48.295408   15284 round_trippers.go:408] Response Status:  in 21008 milliseconds
I0309 12:15:48.295408   15284 kubernetes.go:123] error getting Pods with label selector "component=kube-apiserver" [Get https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 192.168.1.154:8443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
I0309 12:15:48.779136   15284 round_trippers.go:383] GET https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver
I0309 12:15:48.779136   15284 round_trippers.go:390] Request Headers:
I0309 12:15:48.784132   15284 round_trippers.go:393]     User-Agent: minikube.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0309 12:15:48.789189   15284 round_trippers.go:393]     Accept: application/json, */*
I0309 12:16:09.821472   15284 round_trippers.go:408] Response Status:  in 21030 milliseconds
I0309 12:16:09.821792   15284 kubernetes.go:123] error getting Pods with label selector "component=kube-apiserver" [Get https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 192.168.1.154:8443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
I0309 12:16:10.273950   15284 round_trippers.go:383] GET https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver
I0309 12:16:10.273950   15284 round_trippers.go:390] Request Headers:
I0309 12:16:10.279934   15284 round_trippers.go:393]     Accept: application/json, */*
I0309 12:16:10.286962   15284 round_trippers.go:393]     User-Agent: minikube.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0309 12:16:31.326301   15284 round_trippers.go:408] Response Status:  in 21029 milliseconds
I0309 12:16:31.326301   15284 kubernetes.go:123] error getting Pods with label selector "component=kube-apiserver" [Get https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 192.168.1.154:8443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
I0309 12:16:31.776062   15284 round_trippers.go:383] GET https://192.168.1.154:8443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver
I0309 12:16:31.776062   15284 round_trippers.go:390] Request Headers:
I0309 12:16:31.778067   15284 round_trippers.go:393]     Accept: application/json, */*
I0309 12:16:31.779058   15284 round_trippers.go:393]     User-Agent: minikube.exe/v0.0.0 (windows/amd64) kubernetes/$Format

status:

PS C:\WINDOWS\system32> minikube status
host: Running
kubelet: Running
apiserver: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.1.154
PS C:\WINDOWS\system32>

this is the output of the logs

Mar 09 18:15:06 minikube kubelet[3041]: E0309 18:15:06.520150    3041 pod_workers.go:190] Error syncing pod 5c72fb06dcdda608211b70d63c0ca488 ("kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"), skipping: failed to "StartContainer" for "kube-addon-manager" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-addon-manager pod=kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"
Mar 09 18:15:07 minikube kubelet[3041]: E0309 18:15:07.241758    3041 pod_workers.go:190] Error syncing pod 5c72fb06dcdda608211b70d63c0ca488 ("kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"), skipping: failed to "StartContainer" for "kube-addon-manager" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-addon-manager pod=kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"
Mar 09 18:15:07 minikube kubelet[3041]: E0309 18:15:07.297540    3041 pod_workers.go:190] Error syncing pod 479ee245-4293-11e9-a294-00155d380103 ("storage-provisioner_kube-system(479ee245-4293-11e9-a294-00155d380103)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "Back-off 40s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(479ee245-4293-11e9-a294-00155d380103)"
Mar 09 18:15:08 minikube kubelet[3041]: E0309 18:15:08.346557    3041 pod_workers.go:190] Error syncing pod 5c72fb06dcdda608211b70d63c0ca488 ("kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"), skipping: failed to "StartContainer" for "kube-addon-manager" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-addon-manager pod=kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"
Mar 09 18:15:20 minikube kubelet[3041]: E0309 18:15:20.086092    3041 pod_workers.go:190] Error syncing pod 5c72fb06dcdda608211b70d63c0ca488 ("kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"), skipping: failed to "StartContainer" for "kube-addon-manager" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-addon-manager pod=kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488)"
Mar 09 18:15:21 minikube kubelet[3041]: E0309 18:15:21.086822    3041 pod_workers.go:190] Error syncing pod 479ee245-4293-11e9-a294-00155d380103 ("storage-provisioner_kube-system(479ee245-4293-11e9-a294-00155d380103)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "Back-off 40s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(479ee245-4293-11e9-a294-00155d380103)"
Mar 09 18:15:36 minikube kubelet[3041]: E0309 18:15:36.085431    3041 pod_workers.go:190] Error syncing pod 479ee245-4293-11e9-a294-00155d380103 ("storage-provisioner_kube-system(479ee245-4293-11e9-a294-00155d380103)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "Back-off 40s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(479ee245-4293-11e9-a294-00155d380103)"
PS C:\WINDOWS\system32>

@anvvishnu
Copy link
Author

I tried using both an internal switch and an external switch created in Hyper V but it is not working

@anvvishnu
Copy link
Author

I am using a Windows 10 machine, minikube version v.35.0

@tstromberg
Copy link
Contributor

tstromberg commented Mar 9, 2019 •
edited
Loading

So, the apiserver crashed. Can you drag & drop the output of the minikube logs command here? It should include a section for the apiserver logs. Alternatively, you can run this to get the apiserver log:

minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'

This may be related to #1766 - where strange things happen when dynamic memory is enabled in HyperV. If you get a chance, do you mind trying to start the HyperV Manager, and unchecking the Enable Dynamic Memory checkbox for the minikube VM?

https://www.veeam.com/blog/hyper-v-dynamic-memory-managing-vm-ram.html

I don't have a HyperV host yet, otherwise I would give better directions. Please let me know if this helps. Thank you again for the detailed information!

@tstromberg tstromberg added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Mar 13, 2019
@ngrigoriev
Copy link

ngrigoriev commented Mar 16, 2019 •
edited
Loading

Same problem with the latest version on Mac. I have deleted everything, including ~/.minikube and retried from scratch. I am getting consistent results:

$ minikube start
o   minikube v0.35.0 on darwin (amd64)
>   Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
@   Downloading Minikube ISO ...
 184.42 MB / 184.42 MB [============================================] 100.00% 0s
-   "minikube" IP address is 192.168.99.100
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
@   Downloading kubelet v1.13.4
@   Downloading kubeadm v1.13.4

-   Pulling images required by Kubernetes v1.13.4 ...
-   Launching Kubernetes v1.13.4 using kubeadm ...
:   Waiting for pods: apiserver!   Error starting cluster: wait: waiting for component=kube-apiserver: timed out waiting for the condition

*   Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
-   https://github.com/kubernetes/minikube/issues/new

Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0316 14:34:31.153862       1 server.go:557] external host was not specified, using 192.168.99.100
I0316 14:34:31.153963       1 server.go:146] Version: v1.13.4
I0316 14:34:31.734166       1 initialization.go:91] enabled Initializers feature as part of admission plugin setup
I0316 14:34:31.739120       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0316 14:34:31.739162       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0316 14:34:31.741302       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0316 14:34:31.741382       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0316 14:34:33.050740       1 master.go:228] Using reconciler: lease
W0316 14:34:34.551370       1 genericapiserver.go:338] Skipping API batch/v2alpha1 because it has no resources.
W0316 14:34:34.680259       1 genericapiserver.go:338] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0316 14:34:34.687684       1 genericapiserver.go:338] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0316 14:34:34.712917       1 genericapiserver.go:338] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0316 14:34:34.764695       1 genericapiserver.go:338] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2019/03/16 14:34:34 log.go:33: [restful/swagger] listing is available at https://192.168.99.100:8443/swaggerapi
[restful] 2019/03/16 14:34:34 log.go:33: [restful/swagger] https://192.168.99.100:8443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2019/03/16 14:34:36 log.go:33: [restful/swagger] listing is available at https://192.168.99.100:8443/swaggerapi
[restful] 2019/03/16 14:34:36 log.go:33: [restful/swagger] https://192.168.99.100:8443/swaggerui/ is mapped to folder /swagger-ui/
I0316 14:34:36.454935       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0316 14:34:36.455239       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0316 14:34:39.406266       1 secure_serving.go:116] Serving securely on [::]:8443
I0316 14:34:39.406359       1 crd_finalizer.go:242] Starting CRDFinalizer
I0316 14:34:39.406501       1 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0316 14:34:39.406523       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0316 14:34:39.406541       1 autoregister_controller.go:136] Starting autoregister controller
I0316 14:34:39.406544       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0316 14:34:39.407277       1 available_controller.go:283] Starting AvailableConditionController
I0316 14:34:39.407283       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0316 14:34:39.407292       1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0316 14:34:39.407297       1 controller_utils.go:1027] Waiting for caches to sync for crd-autoregister controller
I0316 14:34:39.409508       1 controller.go:84] Starting OpenAPI AggregationController
I0316 14:34:39.410200       1 customresource_discovery_controller.go:203] Starting DiscoveryController
I0316 14:34:39.410303       1 naming_controller.go:284] Starting NamingConditionController
I0316 14:34:39.410326       1 establishing_controller.go:73] Starting EstablishingController
I0316 14:34:39.607830       1 controller_utils.go:1034] Caches are synced for crd-autoregister controller
I0316 14:34:39.607966       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0316 14:34:39.608007       1 cache.go:39] Caches are synced for autoregister controller
I0316 14:34:39.608417       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0316 14:34:43.553373       1 trace.go:76] Trace[1753362574]: "Create /api/v1/namespaces" (started: 2019-03-16 14:34:39.548607975 +0000 UTC m=+8.553812636) (total time: 4.004738805s):
Trace[1753362574]: [4.002567602s] [4.002543742s] About to store object in database
I0316 14:34:43.619644       1 trace.go:76] Trace[1723676016]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:39.609412913 +0000 UTC m=+8.614617585) (total time: 4.010211129s):
Trace[1723676016]: [4.002169169s] [4.002106715s] About to store object in database
I0316 14:34:43.620298       1 trace.go:76] Trace[524682516]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:39.609282097 +0000 UTC m=+8.614486767) (total time: 4.011003151s):
Trace[524682516]: [4.002585055s] [4.0025546s] About to store object in database
I0316 14:34:43.622768       1 trace.go:76] Trace[869159645]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:39.609225654 +0000 UTC m=+8.614430325) (total time: 4.013527062s):
Trace[869159645]: [4.002572783s] [4.002514582s] About to store object in database
I0316 14:34:43.622991       1 trace.go:76] Trace[407155077]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:39.609158296 +0000 UTC m=+8.614362966) (total time: 4.013815803s):
Trace[407155077]: [4.002746945s] [4.002690058s] About to store object in database
I0316 14:34:43.623333       1 trace.go:76] Trace[1375009109]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:39.610301806 +0000 UTC m=+8.615506476) (total time: 4.013017278s):
Trace[1375009109]: [4.001220215s] [4.001132001s] About to store object in database
I0316 14:34:43.630175       1 trace.go:76] Trace[446033863]: "Create /api/v1/nodes" (started: 2019-03-16 14:34:39.520896935 +0000 UTC m=+8.526101595) (total time: 4.109257661s):
Trace[446033863]: [4.104782026s] [4.10469741s] About to store object in database
I0316 14:34:44.414322       1 trace.go:76] Trace[1663596148]: "Create /apis/scheduling.k8s.io/v1beta1/priorityclasses" (started: 2019-03-16 14:34:40.409441386 +0000 UTC m=+9.414646055) (total time: 4.004854961s):
Trace[1663596148]: [4.002747449s] [4.002717146s] About to store object in database
I0316 14:34:44.414797       1 storage_scheduling.go:91] created PriorityClass system-node-critical with value 2000001000
I0316 14:34:44.417358       1 trace.go:76] Trace[1921064204]: "Create /api/v1/namespaces" (started: 2019-03-16 14:34:40.412621437 +0000 UTC m=+9.417826110) (total time: 4.004716203s):
Trace[1921064204]: [4.002686208s] [4.002658783s] About to store object in database
I0316 14:34:44.431738       1 trace.go:76] Trace[505638771]: "Create /apis/rbac.authorization.k8s.io/v1/clusterroles" (started: 2019-03-16 14:34:40.422336919 +0000 UTC m=+9.427541587) (total time: 4.00937661s):
Trace[505638771]: [4.002754836s] [4.00272422s] About to store object in database
I0316 14:34:44.432224       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/cluster-admin
I0316 14:34:46.297303       1 trace.go:76] Trace[1310875407]: "Create /api/v1/namespaces" (started: 2019-03-16 14:34:42.288799812 +0000 UTC m=+11.294004472) (total time: 4.008483333s):
Trace[1310875407]: [4.001854078s] [4.001296855s] About to store object in database
I0316 14:34:47.564919       1 trace.go:76] Trace[837723045]: "Create /api/v1/namespaces/default/services" (started: 2019-03-16 14:34:43.555398552 +0000 UTC m=+12.560603222) (total time: 4.009503322s):
Trace[837723045]: [4.002922269s] [4.002889741s] About to store object in database
W0316 14:34:47.569712       1 lease.go:222] Resetting endpoints for master service "kubernetes" to [192.168.99.100]
I0316 14:34:47.570459       1 controller.go:608] quota admission added evaluator for: endpoints
I0316 14:34:47.633192       1 trace.go:76] Trace[826643205]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:43.624896673 +0000 UTC m=+12.630101343) (total time: 4.008273591s):
Trace[826643205]: [4.003613661s] [4.003345571s] About to store object in database
I0316 14:34:47.638590       1 trace.go:76] Trace[117419162]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:43.621530418 +0000 UTC m=+12.626735088) (total time: 4.017045113s):
Trace[117419162]: [4.007276364s] [4.00723877s] About to store object in database
I0316 14:34:47.641128       1 trace.go:76] Trace[290669117]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:43.621542746 +0000 UTC m=+12.626747416) (total time: 4.01956525s):
Trace[290669117]: [4.007504802s] [4.007345717s] About to store object in database
I0316 14:34:47.641452       1 trace.go:76] Trace[781308168]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:43.624109899 +0000 UTC m=+12.629314569) (total time: 4.017326143s):
Trace[781308168]: [4.004774984s] [4.004741032s] About to store object in database
I0316 14:34:47.641665       1 trace.go:76] Trace[804851517]: "Create /apis/apiregistration.k8s.io/v1/apiservices" (started: 2019-03-16 14:34:43.624236755 +0000 UTC m=+12.629441426) (total time: 4.017416138s):
Trace[804851517]: [4.00439745s] [4.003804792s] About to store object in database
I0316 14:34:47.852310       1 trace.go:76] Trace[191164192]: "Create /api/v1/namespaces/default/events" (started: 2019-03-16 14:34:43.845795116 +0000 UTC m=+12.850999777) (total time: 4.006489469s):
Trace[191164192]: [4.003566446s] [4.003526815s] About to store object in database
I0316 14:34:48.425325       1 trace.go:76] Trace[2023756413]: "Create /api/v1/namespaces/kube-system/configmaps" (started: 2019-03-16 14:34:44.419678964 +0000 UTC m=+13.424883634) (total time: 4.005625812s):
Trace[2023756413]: [4.002024317s] [4.001977037s] About to store object in database
I0316 14:34:48.429837       1 trace.go:76] Trace[1714523224]: "Create /apis/scheduling.k8s.io/v1beta1/priorityclasses" (started: 2019-03-16 14:34:44.42494106 +0000 UTC m=+13.430145731) (total time: 4.004871897s):
Trace[1714523224]: [4.003344376s] [4.003313291s] About to store object in database
I0316 14:34:48.430406       1 storage_scheduling.go:91] created PriorityClass system-cluster-critical with value 2000000000
I0316 14:34:48.430465       1 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
I0316 14:34:48.438195       1 trace.go:76] Trace[1517140422]: "Create /apis/rbac.authorization.k8s.io/v1/clusterroles" (started: 2019-03-16 14:34:44.434069383 +0000 UTC m=+13.439274051) (total time: 4.004098102s):
Trace[1517140422]: [4.001743889s] [4.001720391s] About to store object in database
I0316 14:34:48.438803       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:discovery
I0316 14:34:48.446486       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:basic-user
I0316 14:34:48.449478       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/admin
I0316 14:34:48.452916       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/edit
I0316 14:34:48.457565       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/view
I0316 14:34:48.461009       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin
I0316 14:34:48.464280       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-edit
I0316 14:34:48.467316       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-view
I0316 14:34:48.471182       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:heapster
I0316 14:34:48.474346       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node
I0316 14:34:48.477178       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-problem-detector
I0316 14:34:48.480020       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-proxier
I0316 14:34:48.483368       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kubelet-api-admin
I0316 14:34:48.487057       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-bootstrapper
I0316 14:34:48.489516       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:auth-delegator
I0316 14:34:48.492321       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-aggregator
I0316 14:34:48.495259       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-controller-manager
I0316 14:34:48.499031       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-scheduler
I0316 14:34:48.501817       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-dns
I0316 14:34:48.504217       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:persistent-volume-provisioner
I0316 14:34:48.506835       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:csi-external-attacher
I0316 14:34:48.510178       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aws-cloud-provider
I0316 14:34:48.513792       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient
I0316 14:34:48.517436       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
I0316 14:34:48.520722       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:volume-scheduler
I0316 14:34:48.525626       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:csi-external-provisioner
I0316 14:34:48.530284       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:attachdetach-controller
I0316 14:34:48.535054       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
I0316 14:34:48.539517       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:cronjob-controller
I0316 14:34:48.545403       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:daemon-set-controller
I0316 14:34:48.549008       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:deployment-controller
I0316 14:34:48.553871       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:disruption-controller
I0316 14:34:48.557407       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:endpoint-controller
I0316 14:34:48.560648       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:expand-controller
I0316 14:34:48.564078       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
I0316 14:34:48.566619       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
I0316 14:34:48.569968       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:job-controller
I0316 14:34:48.573476       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:namespace-controller
I0316 14:34:48.575985       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:node-controller
I0316 14:34:48.579370       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
I0316 14:34:48.582369       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
I0316 14:34:48.585563       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:replicaset-controller
I0316 14:34:48.589695       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:replication-controller
I0316 14:34:48.592350       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:resourcequota-controller
I0316 14:34:48.594769       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:route-controller
I0316 14:34:48.598542       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:service-account-controller
I0316 14:34:48.601490       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:service-controller
I0316 14:34:48.605432       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:statefulset-controller
I0316 14:34:48.608728       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:ttl-controller
I0316 14:34:48.612226       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:certificate-controller
I0316 14:34:48.614901       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
I0316 14:34:48.617833       1 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pv-protection-controller
I0316 14:34:48.621087       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/cluster-admin
I0316 14:34:48.623841       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:discovery
I0316 14:34:48.628115       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:basic-user
I0316 14:34:48.642089       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node-proxier
I0316 14:34:48.683635       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-controller-manager
I0316 14:34:48.723379       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-dns
I0316 14:34:48.762670       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-scheduler
I0316 14:34:48.804561       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:aws-cloud-provider
I0316 14:34:48.843176       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node
I0316 14:34:48.883148       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:volume-scheduler
I0316 14:34:48.923165       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:attachdetach-controller
I0316 14:34:48.964554       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
I0316 14:34:49.003583       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:cronjob-controller
I0316 14:34:49.043019       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:daemon-set-controller
I0316 14:34:49.083375       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:deployment-controller
I0316 14:34:49.124214       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:disruption-controller
I0316 14:34:49.163086       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpoint-controller
I0316 14:34:49.203245       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
I0316 14:34:49.241753       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
I0316 14:34:49.283594       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
I0316 14:34:49.321742       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
I0316 14:34:49.362633       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
I0316 14:34:49.401280       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
I0316 14:34:49.443194       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
I0316 14:34:49.481567       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
I0316 14:34:49.522673       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replicaset-controller
I0316 14:34:49.563916       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replication-controller
I0316 14:34:49.602759       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:resourcequota-controller
I0316 14:34:49.643726       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:route-controller
I0316 14:34:49.682734       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-account-controller
I0316 14:34:49.722425       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-controller
I0316 14:34:49.761857       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:statefulset-controller
I0316 14:34:49.802266       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:ttl-controller
I0316 14:34:49.843597       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:certificate-controller
I0316 14:34:49.886353       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
I0316 14:34:49.922526       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pv-protection-controller
I0316 14:34:49.961967       1 controller.go:608] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0316 14:34:49.964541       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader in kube-system
I0316 14:34:50.003491       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0316 14:34:50.042758       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0316 14:34:50.083132       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0316 14:34:50.123882       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0316 14:34:50.163300       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0316 14:34:50.205540       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0316 14:34:50.240407       1 controller.go:608] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0316 14:34:50.243257       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0316 14:34:50.283591       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0316 14:34:50.323480       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0316 14:34:50.362873       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0316 14:34:50.402993       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0316 14:34:50.443968       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0316 14:34:51.464843       1 controller.go:608] quota admission added evaluator for: serviceaccounts
I0316 14:34:51.655156       1 controller.go:608] quota admission added evaluator for: deployments.apps
I0316 14:34:51.702704       1 controller.go:608] quota admission added evaluator for: daemonsets.apps
I0316 14:34:56.521855       1 controller.go:608] quota admission added evaluator for: namespaces
I0316 14:34:57.957194       1 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
I0316 14:34:58.003293       1 controller.go:608] quota admission added evaluator for: replicasets.apps


$ minikube logs
==> coredns <==
.:53
2019-03-16T14:35:04.44Z [INFO] CoreDNS-1.2.6
2019-03-16T14:35:04.44Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
E0316 14:35:29.440539       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:318: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0316 14:35:29.442167       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:311: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0316 14:35:29.442252       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:313: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
==> kube-apiserver <==
I0316 14:34:48.804561       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:aws-cloud-provider
I0316 14:34:48.843176       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node
I0316 14:34:48.883148       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:volume-scheduler
I0316 14:34:48.923165       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:attachdetach-controller
I0316 14:34:48.964554       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
I0316 14:34:49.003583       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:cronjob-controller
I0316 14:34:49.043019       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:daemon-set-controller
I0316 14:34:49.083375       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:deployment-controller
I0316 14:34:49.124214       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:disruption-controller
I0316 14:34:49.163086       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpoint-controller
I0316 14:34:49.203245       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
I0316 14:34:49.241753       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
I0316 14:34:49.283594       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
I0316 14:34:49.321742       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
I0316 14:34:49.362633       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
I0316 14:34:49.401280       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
I0316 14:34:49.443194       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
I0316 14:34:49.481567       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
I0316 14:34:49.522673       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replicaset-controller
I0316 14:34:49.563916       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replication-controller
I0316 14:34:49.602759       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:resourcequota-controller
I0316 14:34:49.643726       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:route-controller
I0316 14:34:49.682734       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-account-controller
I0316 14:34:49.722425       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-controller
I0316 14:34:49.761857       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:statefulset-controller
I0316 14:34:49.802266       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:ttl-controller
I0316 14:34:49.843597       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:certificate-controller
I0316 14:34:49.886353       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
I0316 14:34:49.922526       1 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pv-protection-controller
I0316 14:34:49.961967       1 controller.go:608] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0316 14:34:49.964541       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader in kube-system
I0316 14:34:50.003491       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0316 14:34:50.042758       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0316 14:34:50.083132       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0316 14:34:50.123882       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0316 14:34:50.163300       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0316 14:34:50.205540       1 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0316 14:34:50.240407       1 controller.go:608] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0316 14:34:50.243257       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0316 14:34:50.283591       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0316 14:34:50.323480       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0316 14:34:50.362873       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0316 14:34:50.402993       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0316 14:34:50.443968       1 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0316 14:34:51.464843       1 controller.go:608] quota admission added evaluator for: serviceaccounts
I0316 14:34:51.655156       1 controller.go:608] quota admission added evaluator for: deployments.apps
I0316 14:34:51.702704       1 controller.go:608] quota admission added evaluator for: daemonsets.apps
I0316 14:34:56.521855       1 controller.go:608] quota admission added evaluator for: namespaces
I0316 14:34:57.957194       1 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
I0316 14:34:58.003293       1 controller.go:608] quota admission added evaluator for: replicasets.apps
==> kube-scheduler <==
E0316 14:34:44.529354       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0316 14:34:44.530808       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0316 14:34:44.534366       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0316 14:34:44.535192       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0316 14:34:44.543822       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0316 14:34:44.544828       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0316 14:34:45.525677       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0316 14:34:45.529640       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0316 14:34:45.529986       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0316 14:34:45.530582       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0316 14:34:45.531521       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0316 14:34:45.532955       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0316 14:34:45.535681       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0316 14:34:45.536883       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0316 14:34:45.544868       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0316 14:34:45.546169       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0316 14:34:46.528575       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0316 14:34:46.532160       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0316 14:34:46.532160       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0316 14:34:46.534369       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0316 14:34:46.534388       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0316 14:34:46.535475       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0316 14:34:46.536520       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0316 14:34:46.537918       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0316 14:34:46.546702       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0316 14:34:46.547742       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0316 14:34:47.532989       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0316 14:34:47.533041       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0316 14:34:47.534288       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0316 14:34:47.537443       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0316 14:34:47.538911       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0316 14:34:47.539001       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0316 14:34:47.540005       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0316 14:34:47.540687       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0316 14:34:47.548113       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0316 14:34:47.548844       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0316 14:34:48.534037       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0316 14:34:48.535375       1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0316 14:34:48.537384       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0316 14:34:48.538102       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0316 14:34:48.540054       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0316 14:34:48.541380       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0316 14:34:48.542520       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0316 14:34:48.544023       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0316 14:34:48.549446       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0316 14:34:48.550899       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
I0316 14:34:50.358013       1 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
I0316 14:34:50.458308       1 controller_utils.go:1034] Caches are synced for scheduler controller
I0316 14:34:50.458352       1 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-scheduler...
I0316 14:34:50.465331       1 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler
==> kubelet <==
-- Logs begin at Sat 2019-03-16 14:28:52 UTC, end at Sat 2019-03-16 14:48:09 UTC. --
Mar 16 14:34:40 minikube kubelet[3213]: E0316 14:34:40.902593    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e603b8b", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87e98b, ext:190535626, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d7099e9b1, ext:242046967, loc:(*time.Location)(0x71d6440)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:40 minikube kubelet[3213]: E0316 14:34:40.990590    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.091068    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.097681    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e60223d", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87d03d, ext:190529148, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d709bacc1, ext:242162439, loc:(*time.Location)(0x71d6440)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.191207    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.291438    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.391948    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.492432    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.497901    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e603307", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87e107, ext:190533449, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d709bbd92, ext:242166743, loc:(*time.Location)(0x71d6440)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.592668    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.692900    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.793313    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.894022    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.899147    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e603b8b", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87e98b, ext:190535626, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d709bc742, ext:242169224, loc:(*time.Location)(0x71d6440)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:41 minikube kubelet[3213]: E0316 14:34:41.994376    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.094490    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.195053    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.295421    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.298439    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e60223d", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87d03d, ext:190529148, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d70db2115, ext:246320981, loc:(*time.Location)(0x71d6440)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.396112    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: I0316 14:34:42.416097    3213 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.496880    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.597471    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.697825    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.698513    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e603307", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87e107, ext:190533449, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d70db2fd4, ext:246324757, loc:(*time.Location)(0x71d6440)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.798143    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:42 minikube kubelet[3213]: E0316 14:34:42.899681    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.000525    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.100417    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e603b8b", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87e98b, ext:190535626, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d70db37ac, ext:246326765, loc:(*time.Location)(0x71d6440)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.100820    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.203060    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.303478    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.404274    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: I0316 14:34:43.420457    3213 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.500849    3213 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.158c770d1e60223d", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d6d87d03d, ext:190529148, loc:(*time.Location)(0x71d6440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf1b603d71274be5, ext:251312680, loc:(*time.Location)(0x71d6440)}}, Count:5, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.505215    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: E0316 14:34:43.605586    3213 kubelet.go:2266] node "minikube" not found
Mar 16 14:34:43 minikube kubelet[3213]: I0316 14:34:43.630895    3213 kubelet_node_status.go:75] Successfully registered node minikube
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.088726    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/ac4c5114-47f8-11e9-902c-0800271edc09-kube-proxy") pod "kube-proxy-d26jq" (UID: "ac4c5114-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.088807    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/ac4c5114-47f8-11e9-902c-0800271edc09-xtables-lock") pod "kube-proxy-d26jq" (UID: "ac4c5114-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.088961    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/ac4c5114-47f8-11e9-902c-0800271edc09-lib-modules") pod "kube-proxy-d26jq" (UID: "ac4c5114-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.088982    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-dg9c6" (UniqueName: "kubernetes.io/secret/ac4c5114-47f8-11e9-902c-0800271edc09-kube-proxy-token-dg9c6") pod "kube-proxy-d26jq" (UID: "ac4c5114-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.189550    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-xt6dz" (UniqueName: "kubernetes.io/secret/ac55d433-47f8-11e9-902c-0800271edc09-coredns-token-xt6dz") pod "coredns-86c58d9df4-2bjvl" (UID: "ac55d433-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.189715    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/ac53cc94-47f8-11e9-902c-0800271edc09-config-volume") pod "coredns-86c58d9df4-p2n5h" (UID: "ac53cc94-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.189774    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-xt6dz" (UniqueName: "kubernetes.io/secret/ac53cc94-47f8-11e9-902c-0800271edc09-coredns-token-xt6dz") pod "coredns-86c58d9df4-p2n5h" (UID: "ac53cc94-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: I0316 14:34:58.189883    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/ac55d433-47f8-11e9-902c-0800271edc09-config-volume") pod "coredns-86c58d9df4-2bjvl" (UID: "ac55d433-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:58 minikube kubelet[3213]: W0316 14:34:58.761525    3213 pod_container_deletor.go:75] Container "280ca5cec063e778ae1b5ce71f37b32c38de2a030d9aac259928cc5a24564c0c" not found in pod's containers
Mar 16 14:34:58 minikube kubelet[3213]: W0316 14:34:58.844520    3213 pod_container_deletor.go:75] Container "322ef047a9ef6a4336f59b169aa52722ab62cd89494a75283f36499853c47474" not found in pod's containers
Mar 16 14:34:59 minikube kubelet[3213]: I0316 14:34:59.720245    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/ad3c3c21-47f8-11e9-902c-0800271edc09-tmp") pod "storage-provisioner" (UID: "ad3c3c21-47f8-11e9-902c-0800271edc09")
Mar 16 14:34:59 minikube kubelet[3213]: I0316 14:34:59.720783    3213 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-629gp" (UniqueName: "kubernetes.io/secret/ad3c3c21-47f8-11e9-902c-0800271edc09-storage-provisioner-token-629gp") pod "storage-provisioner" (UID: "ad3c3c21-47f8-11e9-902c-0800271edc09")


$ minikube status
$ minikube status
host: Running
kubelet: Running
apiserver: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

I have noticed that minikube VM has two network interfaces. Is that normal?

@ezamorad
Copy link

ezamorad commented Apr 26, 2019 •
edited
Loading

Hi. I'm experiencing the same issue on Mac Mohave:

$ minikube start
😄 minikube v1.0.0 on darwin (amd64)
🤹 Downloading Kubernetes v1.14.0 images in the background ...
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿 Downloading Minikube ISO ...
142.88 MB / 142.88 MB [============================================] 100.00% 0s
📶 "minikube" IP address is 192.168.99.102
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.2-ce
⌛ Waiting for image downloads to complete ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubelet v1.14.0
💾 Downloading kubeadm v1.14.0
🚜 Pulling images required by Kubernetes v1.14.0 ...
🚀 Launching Kubernetes v1.14.0 using kubeadm ...
⌛ Waiting for pods: apiserver
💣 Error starting cluster: wait: waiting for component=kube-apiserver: timed out waiting for the condition

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new
❌ Problems detected in "kube-addon-manager":
error: unable to recognize "STDIN": Get https://localhost:8443/api?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused
error: no objects passed to apply
error: no objects passed to apply

$minikube status
host: Running
kubelet: Running
apiserver: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.102

$ minikube logs
==> coredns <==
.:53
2019-04-25T23:49:37.315Z [INFO] CoreDNS-1.3.1
2019-04-25T23:49:37.315Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-04-25T23:49:37.315Z [INFO] plugin/reload: Running configuration MD5 = 599b9eb76b8c147408aed6a0bbe0f669

==> dmesg <==
[ +5.003187] hpet1: lost 318 rtc interrupts
[ +5.002490] hpet1: lost 318 rtc interrupts
[ +5.002040] hpet1: lost 318 rtc interrupts
[ +5.000716] hpet1: lost 318 rtc interrupts
[ +5.001532] hpet1: lost 319 rtc interrupts
[ +5.002062] hpet1: lost 318 rtc interrupts
[ +5.000486] hpet1: lost 318 rtc interrupts
[Apr26 00:10] hpet1: lost 318 rtc interrupts
[ +5.001973] hpet1: lost 318 rtc interrupts
[ +5.000997] hpet1: lost 318 rtc interrupts
[ +5.002216] hpet1: lost 318 rtc interrupts
[ +5.002539] hpet1: lost 318 rtc interrupts
[ +5.001457] hpet1: lost 318 rtc interrupts
[ +5.002304] hpet1: lost 319 rtc interrupts
[ +5.001016] hpet1: lost 318 rtc interrupts
[ +5.001536] hpet1: lost 318 rtc interrupts
[ +5.000908] hpet1: lost 318 rtc interrupts
[ +5.002082] hpet1: lost 318 rtc interrupts
[ +4.999607] hpet1: lost 318 rtc interrupts
[Apr26 00:11] hpet1: lost 318 rtc interrupts
[ +5.001615] hpet1: lost 318 rtc interrupts
[ +5.000796] hpet1: lost 318 rtc interrupts
[ +5.001721] hpet1: lost 318 rtc interrupts
[ +5.001621] hpet1: lost 318 rtc interrupts
[ +5.002598] hpet1: lost 319 rtc interrupts
[ +5.002737] hpet1: lost 318 rtc interrupts
[ +5.001100] hpet1: lost 319 rtc interrupts
[ +5.001447] hpet1: lost 318 rtc interrupts
[ +5.001177] hpet1: lost 319 rtc interrupts
[ +5.002946] hpet1: lost 318 rtc interrupts
[ +4.999778] hpet1: lost 318 rtc interrupts
[Apr26 00:12] hpet1: lost 318 rtc interrupts
[ +5.004699] hpet1: lost 319 rtc interrupts
[ +5.001441] hpet1: lost 318 rtc interrupts
[ +5.001598] hpet1: lost 318 rtc interrupts
[ +5.002597] hpet1: lost 318 rtc interrupts
[ +5.001136] hpet1: lost 318 rtc interrupts
[ +5.002056] hpet1: lost 318 rtc interrupts
[ +5.001490] hpet1: lost 318 rtc interrupts
[ +5.001344] hpet1: lost 318 rtc interrupts
[ +5.001071] hpet1: lost 318 rtc interrupts
[ +5.002148] hpet1: lost 319 rtc interrupts
[ +5.000504] hpet1: lost 318 rtc interrupts
[Apr26 00:13] hpet1: lost 318 rtc interrupts
[ +5.001753] hpet1: lost 318 rtc interrupts
[ +5.003188] hpet1: lost 318 rtc interrupts
[ +5.001363] hpet1: lost 318 rtc interrupts
[ +5.003245] hpet1: lost 318 rtc interrupts
[ +5.001106] hpet1: lost 319 rtc interrupts
[ +5.002203] hpet1: lost 319 rtc interrupts

==> kernel <==
00:13:36 up 27 min, 0 users, load average: 0.21, 0.34, 0.29
Linux minikube 4.15.0 #1 SMP Tue Mar 26 02:53:14 UTC 2019 x86_64 GNU/Linux

==> kube-addon-manager <==
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:06:08+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:07:06+00:00 ==
INFO: == Reconciling with deprecated label ==
error: no objects passed to apply
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:07:08+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:08:06+00:00 ==
INFO: == Reconciling with deprecated label ==
error: no objects passed to apply
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:08:07+00:00 ==
INFO: Leader is minikube
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:09:07+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:09:09+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:10:06+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:10:07+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:11:06+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:11:08+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:12:06+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:12:08+00:00 ==
INFO: Leader is minikube
INFO: == Kubernetes addon ensure completed at 2019-04-26T00:13:06+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-04-26T00:13:07+00:00 ==

==> kube-apiserver <==
I0426 00:13:12.856065 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:12.856214 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:13.856885 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:13.857356 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:14.858021 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:14.858209 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:15.858580 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:15.858995 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:16.859418 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:16.859920 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:17.860566 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:17.860840 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:18.861987 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:18.862170 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:19.863157 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:19.863313 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:20.863764 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:20.863900 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:21.864801 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:21.865439 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:22.866421 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:22.866531 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:23.867563 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:23.867944 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:24.869069 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:24.869180 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:25.870462 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:25.870896 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:26.872493 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:26.873029 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:27.873584 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:27.874031 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:28.874323 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:28.874990 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:29.875360 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:29.875473 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:30.875754 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:30.875864 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:31.876380 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:31.876699 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:32.877541 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:32.877800 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:33.878684 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:33.878960 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:34.879498 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:34.879827 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:35.880825 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:35.881099 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002
I0426 00:13:36.881386 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001
I0426 00:13:36.881480 1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000002

==> kube-proxy <==
W0425 23:49:07.661835 1 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
I0425 23:49:07.672582 1 server_others.go:148] Using iptables Proxier.
W0425 23:49:07.672832 1 proxier.go:319] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0425 23:49:07.673301 1 server_others.go:178] Tearing down inactive rules.
I0425 23:49:07.851102 1 server.go:555] Version: v1.14.0
I0425 23:49:07.864647 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0425 23:49:07.864689 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0425 23:49:07.865046 1 conntrack.go:83] Setting conntrack hashsize to 32768
I0425 23:49:07.872030 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0425 23:49:07.872149 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0425 23:49:07.872269 1 config.go:202] Starting service config controller
I0425 23:49:07.872293 1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0425 23:49:07.872380 1 config.go:102] Starting endpoints config controller
I0425 23:49:07.872405 1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0425 23:49:07.972758 1 controller_utils.go:1034] Caches are synced for endpoints config controller
I0425 23:49:07.972765 1 controller_utils.go:1034] Caches are synced for service config controller

==> kube-scheduler <==
I0425 23:48:50.352884 1 serving.go:319] Generated self-signed cert in-memory
W0425 23:48:51.285427 1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0425 23:48:51.285454 1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0425 23:48:51.285464 1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0425 23:48:51.296220 1 server.go:142] Version: v1.14.0
I0425 23:48:51.296269 1 defaults.go:87] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W0425 23:48:51.297103 1 authorization.go:47] Authorization is disabled
W0425 23:48:51.297125 1 authentication.go:55] Authentication is disabled
I0425 23:48:51.297136 1 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
I0425 23:48:51.297583 1 secure_serving.go:116] Serving securely on 127.0.0.1:10259
E0425 23:48:55.464576 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0425 23:48:55.464592 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0425 23:48:55.464857 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0425 23:48:55.464967 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0425 23:48:55.465076 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0425 23:48:55.465193 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0425 23:48:55.465412 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0425 23:48:55.465429 1 reflector.go:126] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:223: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0425 23:48:55.465447 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0425 23:48:55.468384 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0425 23:48:56.465631 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0425 23:48:56.467365 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0425 23:48:56.469594 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0425 23:48:56.471127 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0425 23:48:56.474092 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0425 23:48:56.474092 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0425 23:48:56.475921 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0425 23:48:56.477050 1 reflector.go:126] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:223: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0425 23:48:56.481171 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0425 23:48:56.484119 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
I0425 23:48:58.302288 1 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
I0425 23:48:58.402610 1 controller_utils.go:1034] Caches are synced for scheduler controller
I0425 23:48:58.402956 1 leaderelection.go:217] attempting to acquire leader lease kube-system/kube-scheduler...
I0425 23:48:58.410690 1 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler

==> kubelet <==
-- Logs begin at Thu 2019-04-25 23:46:44 UTC, end at Fri 2019-04-26 00:13:36 UTC. --
Apr 25 23:48:53 minikube kubelet[3202]: I0425 23:48:53.451007 3202 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
Apr 25 23:48:53 minikube kubelet[3202]: I0425 23:48:53.453300 3202 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
Apr 25 23:48:53 minikube kubelet[3202]: I0425 23:48:53.454724 3202 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
Apr 25 23:48:53 minikube kubelet[3202]: E0425 23:48:53.491649 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:53 minikube kubelet[3202]: E0425 23:48:53.593221 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:53 minikube kubelet[3202]: E0425 23:48:53.693965 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:53 minikube kubelet[3202]: E0425 23:48:53.794553 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:53 minikube kubelet[3202]: E0425 23:48:53.895054 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:53 minikube kubelet[3202]: E0425 23:48:53.997756 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.098172 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.199204 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.299360 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.399733 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.500196 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.600758 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.701819 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.802013 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:54 minikube kubelet[3202]: E0425 23:48:54.903181 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.004931 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.105630 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.206207 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.306826 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.407721 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.485509 3202 controller.go:194] failed to get node "minikube" when trying to set owner ref to the node lease: nodes "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: E0425 23:48:55.508399 3202 kubelet.go:2244] node "minikube" not found
Apr 25 23:48:55 minikube kubelet[3202]: I0425 23:48:55.559381 3202 kubelet_node_status.go:75] Successfully registered node minikube
Apr 25 23:48:55 minikube kubelet[3202]: I0425 23:48:55.608933 3202 reconciler.go:154] Reconciler: start to sync state
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.225845 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc838200331e", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb825ee731e, ext:169305194, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb825ee731e, ext:169305194, loc:(*time.Location)(0x7ff88e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.280528 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc83875fb776", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4df776, ext:259451074, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4df776, ext:259451074, loc:(*time.Location)(0x7ff88e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.334377 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8387601180", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5180, ext:259474123, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5180, ext:259474123, loc:(*time.Location)(0x7ff88e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.390078 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8387601fae", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5fae, ext:259477753, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5fae, ext:259477753, loc:(*time.Location)(0x7ff88e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.449026 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc83875fb776", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4df776, ext:259451074, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82c36156f, ext:274663098, loc:(*time.Location)(0x7ff88e0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.508277 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8387601180", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5180, ext:259474123, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82c362a17, ext:274668386, loc:(*time.Location)(0x7ff88e0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.566178 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8387601fae", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5fae, ext:259477753, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82c363a63, ext:274672559, loc:(*time.Location)(0x7ff88e0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.622038 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8388b61254", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeAllocatableEnforced", Message:"Updated Node Allocatable limit across pods", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82ca45254, ext:281887642, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82ca45254, ext:281887642, loc:(*time.Location)(0x7ff88e0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:57 minikube kubelet[3202]: E0425 23:48:57.827826 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8387601fae", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node minikube status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5fae, ext:259477753, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb8387760ac, ext:480268800, loc:(*time.Location)(0x7ff88e0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:58 minikube kubelet[3202]: E0425 23:48:58.224540 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc83875fb776", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node minikube status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4df776, ext:259451074, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb83875c498, ext:480163323, loc:(*time.Location)(0x7ff88e0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:48:58 minikube kubelet[3202]: E0425 23:48:58.626478 3202 event.go:191] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"minikube.1598dc8387601180", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"minikube", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node minikube status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb82b4e5180, ext:259474123, loc:(*time.Location)(0x7ff88e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf28afb838775035, ext:480264585, loc:(*time.Location)(0x7ff88e0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!)
Apr 25 23:49:05 minikube kubelet[3202]: E0425 23:49:05.509720 3202 reflector.go:126] object-"kube-system"/"kube-proxy-token-mww24": Failed to list *v1.Secret: secrets "kube-proxy-token-mww24" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "minikube" and this object
Apr 25 23:49:05 minikube kubelet[3202]: E0425 23:49:05.511812 3202 reflector.go:126] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:minikube" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "minikube" and this object
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.572050 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/b5df82f9-67b4-11e9-9559-080027e3367b-config-volume") pod "coredns-fb8b8dccf-ng4dd" (UID: "b5df82f9-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.572100 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-zx5vf" (UniqueName: "kubernetes.io/secret/b5df82f9-67b4-11e9-9559-080027e3367b-coredns-token-zx5vf") pod "coredns-fb8b8dccf-ng4dd" (UID: "b5df82f9-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.672949 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b5e0114b-67b4-11e9-9559-080027e3367b-xtables-lock") pod "kube-proxy-9nnw5" (UID: "b5e0114b-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.673290 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-mww24" (UniqueName: "kubernetes.io/secret/b5e0114b-67b4-11e9-9559-080027e3367b-kube-proxy-token-mww24") pod "kube-proxy-9nnw5" (UID: "b5e0114b-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.673548 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-zx5vf" (UniqueName: "kubernetes.io/secret/b5e5a124-67b4-11e9-9559-080027e3367b-coredns-token-zx5vf") pod "coredns-fb8b8dccf-xzb6f" (UID: "b5e5a124-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.673610 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/b5e0114b-67b4-11e9-9559-080027e3367b-kube-proxy") pod "kube-proxy-9nnw5" (UID: "b5e0114b-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.673630 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/b5e0114b-67b4-11e9-9559-080027e3367b-lib-modules") pod "kube-proxy-9nnw5" (UID: "b5e0114b-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:05 minikube kubelet[3202]: I0425 23:49:05.673646 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/b5e5a124-67b4-11e9-9559-080027e3367b-config-volume") pod "coredns-fb8b8dccf-xzb6f" (UID: "b5e5a124-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:07 minikube kubelet[3202]: I0425 23:49:07.585249 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-qdnnn" (UniqueName: "kubernetes.io/secret/b710a8c8-67b4-11e9-9559-080027e3367b-storage-provisioner-token-qdnnn") pod "storage-provisioner" (UID: "b710a8c8-67b4-11e9-9559-080027e3367b")
Apr 25 23:49:07 minikube kubelet[3202]: I0425 23:49:07.586206 3202 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/b710a8c8-67b4-11e9-9559-080027e3367b-tmp") pod "storage-provisioner" (UID: "b710a8c8-67b4-11e9-9559-080027e3367b")

==> storage-provisioner <==

@tstromberg
Copy link
Contributor

Regrettably, there isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate. I suspect this was due to #1766 - but there are other possibilities as well.

If you run into this issue with minikube v1.0.1+ on hyperv, please feel free to add to this issue.

@tstromberg tstromberg added the triage/obsolete Bugs that no longer occur in the latest stable release label May 14, 2019
This was referenced May 14, 2019
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
co/hyperv HyperV related issues ev/apiserver-timeout timeout talking to the apiserver kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it. triage/obsolete Bugs that no longer occur in the latest stable release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tstromberg @ngrigoriev @ezamorad @anvvishnu