Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minikube start: apiserver unhealthy: <nil>: Error #3522

Closed
prksu opened this issue Jan 10, 2019 · 17 comments
Closed

minikube start: apiserver unhealthy: <nil>: Error #3522

prksu opened this issue Jan 10, 2019 · 17 comments

Comments

@prksu
Copy link

prksu commented Jan 10, 2019

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.32.0

  • OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.32.0.iso"
  • Install tools: -
  • Others: -

What happened:

minikube start, apiserver unhealthy: : Error

$ minikube start
Starting local Kubernetes v1.12.4 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Machine exists, restarting cluster components...
Verifying kubelet health ...
Verifying apiserver health .......................error: Temporary Error: apiserver unhealthy: <nil>: Error
Temporary Error: apiserver unhealthy: <nil>: Error
Temporary Error: apiserver unhealthy: <nil>: Error
Temporary Error: apiserver unhealthy: <nil>: Error
$ minikube status
host: Running
kubelet: Running
apiserver: Error
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

after waiting some minutes

$ minikube status
host: Running
kubelet: Running
apiserver: Running

apiserver become running

What you expected to happen:

minikube should start properly

How to reproduce it (as minimally and precisely as possible):

start minikube after stopped

Output of minikube logs (if applicable):

-- Logs begin at Thu 2019-01-10 04:43:26 UTC, end at Thu 2019-01-10 04:46:47 UTC. --
Jan 10 04:44:38 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jan 10 04:44:38 minikube kubelet[3075]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:38 minikube kubelet[3075]: F0110 04:44:38.923558    3075 server.go:245] unable to load client CA file /var/lib/minikube/certs/ca.crt: open /var/lib/minikube/certs/ca.crt: no such file or directory
Jan 10 04:44:38 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 10 04:44:38 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 10 04:44:49 minikube systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 10 04:44:49 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Jan 10 04:44:49 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 10 04:44:49 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jan 10 04:44:49 minikube kubelet[3171]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.133555    3171 server.go:408] Version: v1.12.4
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.133695    3171 plugins.go:99] No cloud provider specified.
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152037    3171 server.go:667] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152267    3171 container_manager_linux.go:247] container manager verified user specified cgroup-root exists: []
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152337    3171 container_manager_linux.go:252] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152456    3171 container_manager_linux.go:271] Creating device plugin manager: true
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152498    3171 state_mem.go:36] [cpumanager] initializing new in-memory state store
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152746    3171 state_mem.go:84] [cpumanager] updated default cpuset: ""
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152755    3171 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152812    3171 kubelet.go:279] Adding pod path: /etc/kubernetes/manifests
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.152839    3171 kubelet.go:304] Watching apiserver
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.157912    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.158060    3171 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.158121    3171 client.go:104] Start docker client with request timeout=2m0s
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.158983    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.159053    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:49 minikube kubelet[3171]: W0110 04:44:49.160718    3171 docker_service.go:540] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.160746    3171 docker_service.go:236] Hairpin mode set to "hairpin-veth"
Jan 10 04:44:49 minikube kubelet[3171]: W0110 04:44:49.163139    3171 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.164431    3171 docker_service.go:251] Docker cri networking managed by kubernetes.io/no-op
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.170972    3171 docker_service.go:256] Docker Info: &{ID:H6UV:HGP4:TGZM:4HZL:MUH6:R4QV:5OXZ:OLBB:TEXX:52XL:QOY2:R5XB Containers:81 ContainersRunning:22 ContainersPaused:0 ContainersStopped:59 Images:182 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:25 OomKillDisable:true NGoroutines:46 SystemTime:2019-01-10T04:44:49.165340023Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.15.0 OperatingSystem:Buildroot 2018.05 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc4206b3810 NCPU:2 MemTotal:8361787392 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:minikube Labels:[provider=virtualbox] ExperimentalBuild:false ServerVersion:18.06.1-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:468a545b9edcd5932818eb9de8e72413e616e86e Expected:468a545b9edcd5932818eb9de8e72413e616e86e} RuncCommit:{ID:69663f0bd4b60df09991c08812a60108003fa340 Expected:69663f0bd4b60df09991c08812a60108003fa340} InitCommit:{ID:N/A Expected:} SecurityOptions:[name=seccomp,profile=default]}
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.172731    3171 docker_service.go:269] Setting cgroupDriver to cgroupfs
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.179321    3171 kuberuntime_manager.go:197] Container runtime docker initialized, version: 18.06.1-ce, apiVersion: 1.38.0
Jan 10 04:44:49 minikube kubelet[3171]: W0110 04:44:49.179498    3171 probe.go:270] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.180803    3171 server.go:1013] Started kubelet
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.181149    3171 kubelet.go:1287] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.181550    3171 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.181596    3171 status_manager.go:152] Starting to sync pod status with apiserver
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.181629    3171 kubelet.go:1804] Starting kubelet main sync loop.
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.181691    3171 kubelet.go:1821] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.181837    3171 server.go:133] Starting to listen on 0.0.0.0:10250
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.182310    3171 server.go:318] Adding debug handlers to kubelet server.
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.184031    3171 volume_manager.go:248] Starting Kubelet Volume Manager
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.184803    3171 desired_state_of_world_populator.go:130] Desired state populator starts to run
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.185150    3171 event.go:212] Unable to write event: 'Post https://localhost:8443/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8443: connect: connection refused' (may retry after sleeping)
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.290534    3171 kubelet.go:1821] skipping pod synchronization - [container runtime is down]
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.290599    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.290959    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.330336    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.331000    3171 kubelet_node_status.go:70] Attempting to register node minikube
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.332404    3171 kubelet_node_status.go:92] Unable to register node "minikube" with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.334219    3171 cpu_manager.go:155] [cpumanager] starting with none policy
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.334311    3171 cpu_manager.go:156] [cpumanager] reconciling every 10s
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.334354    3171 policy_none.go:42] [cpumanager] none policy: Start
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.391205    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.491015    3171 kubelet.go:1821] skipping pod synchronization - [container runtime is down]
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.491328    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.532652    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.535601    3171 kubelet_node_status.go:70] Attempting to register node minikube
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.535840    3171 kubelet_node_status.go:92] Unable to register node "minikube" with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.591645    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.691965    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.735064    3171 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: W0110 04:44:49.741679    3171 docker_sandbox.go:375] failed to read pod IP from plugin/docker: Couldn't find network status for default/encode-srv-5ddf4bdb54-2jkdh through plugin: invalid network status for
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.792341    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.894876    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.895055    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.936201    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.940088    3171 kubelet_node_status.go:70] Attempting to register node minikube
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.940606    3171 kubelet_node_status.go:92] Unable to register node "minikube" with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:49 minikube kubelet[3171]: E0110 04:44:49.995420    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:49 minikube kubelet[3171]: I0110 04:44:49.995698    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/e1b3e16379a55d4c355fa42bc75eb023-kubeconfig") pod "kube-scheduler-minikube" (UID: "e1b3e16379a55d4c355fa42bc75eb023")
Jan 10 04:44:50 minikube kubelet[3171]: W0110 04:44:50.044985    3171 docker_sandbox.go:375] failed to read pod IP from plugin/docker: Couldn't find network status for default/user-api-77d855846f-nt627 through plugin: invalid network status for
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.095552    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.159543    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.159588    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.166948    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.196467    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.196696    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.197161    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: W0110 04:44:50.202861    3171 status_manager.go:485] Failed to get status for pod "kube-scheduler-minikube_kube-system(e1b3e16379a55d4c355fa42bc75eb023)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-scheduler-minikube: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.296561    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.296628    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/d682efea6fd7d1c11b13f78e8c81af08-addons") pod "kube-addon-manager-minikube" (UID: "d682efea6fd7d1c11b13f78e8c81af08")
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.296648    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/d682efea6fd7d1c11b13f78e8c81af08-kubeconfig") pod "kube-addon-manager-minikube" (UID: "d682efea6fd7d1c11b13f78e8c81af08")
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.355110    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.355563    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: W0110 04:44:50.360970    3171 status_manager.go:485] Failed to get status for pod "kube-addon-manager-minikube_kube-system(d682efea6fd7d1c11b13f78e8c81af08)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-addon-manager-minikube: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.396876    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.397007    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/400930335566057521570dcbaf3dbb0b-etcd-data") pod "etcd-minikube" (UID: "400930335566057521570dcbaf3dbb0b")
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.397038    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/400930335566057521570dcbaf3dbb0b-etcd-certs") pod "etcd-minikube" (UID: "400930335566057521570dcbaf3dbb0b")
Jan 10 04:44:50 minikube kubelet[3171]: W0110 04:44:50.412345    3171 docker_sandbox.go:375] failed to read pod IP from plugin/docker: Couldn't find network status for default/popular-srv-6cc66f6966-bhflw through plugin: invalid network status for
Jan 10 04:44:50 minikube kubelet[3171]: W0110 04:44:50.450304    3171 docker_sandbox.go:375] failed to read pod IP from plugin/docker: Couldn't find network status for default/user-srv-f776bdc9d-t6flx through plugin: invalid network status for
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.497276    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.597550    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.698436    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.739841    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.740170    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.741792    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:50 minikube kubelet[3171]: W0110 04:44:50.744392    3171 status_manager.go:485] Failed to get status for pod "etcd-minikube_kube-system(400930335566057521570dcbaf3dbb0b)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/etcd-minikube: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.745775    3171 kubelet_node_status.go:70] Attempting to register node minikube
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.746091    3171 kubelet_node_status.go:92] Unable to register node "minikube" with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.798724    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.798807    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/cd449a681c0514127b733c5f103919f2-k8s-certs") pod "kube-apiserver-minikube" (UID: "cd449a681c0514127b733c5f103919f2")
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.798843    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/cd449a681c0514127b733c5f103919f2-ca-certs") pod "kube-apiserver-minikube" (UID: "cd449a681c0514127b733c5f103919f2")
Jan 10 04:44:50 minikube kubelet[3171]: I0110 04:44:50.798862    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/cd449a681c0514127b733c5f103919f2-usr-share-ca-certificates") pod "kube-apiserver-minikube" (UID: "cd449a681c0514127b733c5f103919f2")
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.899497    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:50 minikube kubelet[3171]: E0110 04:44:50.999733    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.100034    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.147322    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.147375    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.149728    3171 status_manager.go:485] Failed to get status for pod "kube-apiserver-minikube_kube-system(cd449a681c0514127b733c5f103919f2)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-apiserver-minikube: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.160651    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.161917    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.167381    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.201183    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/62402c9ecb1701a979883b1f71d73a28-kubeconfig") pod "kube-controller-manager-minikube" (UID: "62402c9ecb1701a979883b1f71d73a28")
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.201221    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/62402c9ecb1701a979883b1f71d73a28-usr-share-ca-certificates") pod "kube-controller-manager-minikube" (UID: "62402c9ecb1701a979883b1f71d73a28")
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.201267    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/62402c9ecb1701a979883b1f71d73a28-k8s-certs") pod "kube-controller-manager-minikube" (UID: "62402c9ecb1701a979883b1f71d73a28")
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.201282    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/62402c9ecb1701a979883b1f71d73a28-ca-certs") pod "kube-controller-manager-minikube" (UID: "62402c9ecb1701a979883b1f71d73a28")
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.201401    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.301572    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.355789    3171 pod_container_deletor.go:75] Container "2d7d0ce5026775fd33b7023d0cd12384dc045113c0a87d2c2630be2fc0203a1a" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.355815    3171 pod_container_deletor.go:75] Container "6f37ae7b43e171de5dae250ac7989ef7bf78ca86595a8c7c5ee156cca84db0c6" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.355824    3171 pod_container_deletor.go:75] Container "c47d3f37ad4bbd6fce05600e6c1b6d77bd127bf769a2609f17493352d659c4a9" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.355830    3171 pod_container_deletor.go:75] Container "ea53d3905ed8e8bddef67c8e79fee6b94a83e66413cd90b66ec0d90253a2288a" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.355836    3171 pod_container_deletor.go:75] Container "249406cc0fd23fcbc9dce200c570699306be129ac225101272f8c8ad01e5ccaa" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.355878    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.356024    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.366871    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.367532    3171 status_manager.go:485] Failed to get status for pod "kube-controller-manager-minikube_kube-system(62402c9ecb1701a979883b1f71d73a28)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-controller-manager-minikube: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.370888    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.374529    3171 pod_container_deletor.go:75] Container "f839f70b58aa067e83f3947b2e8118a5cfbbf6c0c6e9892fb1dceef98de335bf" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.374661    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.376363    3171 pod_container_deletor.go:75] Container "fd53c6c00e4d97a622d4e1ea3af98a67730aaf329b2425a5646daf1dc46cc542" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.376464    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.378140    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.379707    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.380997    3171 pod_container_deletor.go:75] Container "991cc675e529e2ddb0c064619bc480a81c74406504fa25e8418f08e9f1706f1b" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.381100    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.382563    3171 pod_container_deletor.go:75] Container "33c625f47ecca38904bba47971e2002a6f3d6c38d427fe747156048f4cd620bf" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.382611    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.383853    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.385320    3171 pod_container_deletor.go:75] Container "0318c2a3ad97c17c90a6fa437feeff65cf4f77fb9c607a92ff5bd975bf2ee837" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.385390    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.386515    3171 pod_container_deletor.go:75] Container "708d5a33f696ba26a1810419cd3981acd3d9b27c1a0bd49b034550502466d6a1" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.386605    3171 pod_container_deletor.go:75] Container "62f01586cfbbac9885600f893c28dc7e1d96506487c343b48e5a642f8bc5dfc2" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.386676    3171 pod_container_deletor.go:75] Container "d8f9f2cebb699dc96870b3b330a4538adbe416c253d50af691ebdf51b62e7d34" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.386723    3171 pod_container_deletor.go:75] Container "b099c1183bbbd624fc79da886fc6de0f84e8e85f69e2712c288ba650bf0a4bfc" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.386758    3171 pod_container_deletor.go:75] Container "f20e3d45f0d80de60625b723b78fb4a6b9276c5e115901971174ad26da5f52fa" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.386824    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.388010    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.389149    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.390237    3171 pod_container_deletor.go:75] Container "796eaf6256e8d8fc0b683bbff2773d43292d35c0a7a1106782c54461c6e5621b" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.390316    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391523    3171 pod_container_deletor.go:75] Container "051f771d6ec1614033e7526aa2589e05551b318ff6001f1f866932fe47f6e142" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391577    3171 pod_container_deletor.go:75] Container "95035233f6c18d71a3a36eaf4eca31f92bd20546f67dfa824b7a549fbb5b6571" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391616    3171 pod_container_deletor.go:75] Container "60a671408a5b5527c55177e579b306ee0808754e5fe27777e7797209f740cf31" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391649    3171 pod_container_deletor.go:75] Container "93cc9bb558ae70ba2e0ebcb39c8098be0cd1a15991ec4f3f5b1ae9946383f060" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391680    3171 pod_container_deletor.go:75] Container "a2cb39d09e471388f2a3785464ec855ef9ca0012f5b4865ce647db7f395cac0f" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391747    3171 pod_container_deletor.go:75] Container "462be840370b3fa8d1ccfb6ee9be13a37d673b25893dcfd98298b5c227b20e6f" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391806    3171 pod_container_deletor.go:75] Container "1bb6d12746301cfca0ed88abee688dff14a5ef7f5af5c8c28f0295d855f73f63" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391845    3171 pod_container_deletor.go:75] Container "79598726adf4486a8062090b0cd68f0499c8afdb748d4574db0b1072c5665ca5" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391883    3171 pod_container_deletor.go:75] Container "558a099efdf8106a6599c326a6096e9f3cfb8f55f8d3a1730011c507208b3e64" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391920    3171 pod_container_deletor.go:75] Container "3e247842c0eb32229a3284adfa26fcc7eb96146cfdc48772f8eb077d9d651913" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391950    3171 pod_container_deletor.go:75] Container "4a173ebf9232869fd80c964181190f1305e0dc9f4f5c4f9a3b1d94072868c16a" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.391981    3171 pod_container_deletor.go:75] Container "2b4e1fcec1a7076e795b0a480b908a4ae1afa0f5e7ae3c0f58a0855ab0661d1e" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392013    3171 pod_container_deletor.go:75] Container "2983119aa507f4bca58b086ff3d82a6ff19c04c6ad1c04ecb6a53345319e9e1c" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392064    3171 pod_container_deletor.go:75] Container "5d6ded8292f48f9616c81922f325695fd8a81c9f7bbf2054e1abdc5a7b205448" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392102    3171 pod_container_deletor.go:75] Container "26f8ef3e796db2740d6b786127e3a46c1fce9177a817fcb720de8d963a76d7c8" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392138    3171 pod_container_deletor.go:75] Container "b434d64a02f5e42fa9a3c81bbd976f6a7cfd23acc35e9e40e0873b1f6e1dbea3" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392169    3171 pod_container_deletor.go:75] Container "86684ac1cb583653109a44d5e815a2d120468008d39c29541c24e86fe441c81b" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392216    3171 pod_container_deletor.go:75] Container "57a145322846efd84551d55053ab6babe42f1637ed7e7dff3379b28ccb1509f2" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392252    3171 pod_container_deletor.go:75] Container "9569ba5bcab1a79f508139d0b7d9c6441b0ebded8f64220f7532c8ff89142367" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392283    3171 pod_container_deletor.go:75] Container "e37268fecab2368d246314bfe69934055217c00cda324759407fd8221f71a1c6" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392313    3171 pod_container_deletor.go:75] Container "0261bcda3dc70dcb2bd7fc0d7e46b626cc3f7fc1c9e8bb2caf985df8d4307dc1" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392349    3171 pod_container_deletor.go:75] Container "cc9d6d34e7f585f3953f7f8b9d8b3d09152e1b8b48478a06f3a60055a527b633" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392387    3171 pod_container_deletor.go:75] Container "cb3bfa1ea7d0e5a543ce2067e73f5150c49ed3c161a4a9ed35015f16ee19cba5" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392423    3171 pod_container_deletor.go:75] Container "f4198e05c834f22a155b5060334ade78be142d401a7b0b09fe4c7da01f522938" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392454    3171 pod_container_deletor.go:75] Container "389718cb139c1c54c3c257ff2f9957f163571232bd4f64cd65ca89f8a7fd98fe" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.392484    3171 pod_container_deletor.go:75] Container "5d5ea0164efdf91690c9d1a3d04780b5947839baaf09a03861542c671c730377" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: I0110 04:44:51.392534    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:51 minikube kubelet[3171]: W0110 04:44:51.393873    3171 pod_container_deletor.go:75] Container "bf0386ee76a2973aa8d9322e5a7789611056e048b46fd6d8669face1efe5a2e0" not found in pod's containers
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.401782    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.486184    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "6660fc8036cfd792e5998de838cb75f5d0e78548aae4bb7d9d040631503323f3" for pod "kube-scheduler-minikube_kube-system(e1b3e16379a55d4c355fa42bc75eb023)" error: rpc error: code = Unknown desc = Error: No such container: 6660fc8036cfd792e5998de838cb75f5d0e78548aae4bb7d9d040631503323f3
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.486326    3171 generic.go:241] PLEG: Ignoring events for pod kube-scheduler-minikube/kube-system: rpc error: code = Unknown desc = Error: No such container: 6660fc8036cfd792e5998de838cb75f5d0e78548aae4bb7d9d040631503323f3
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.502101    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.602454    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.702553    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.802744    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:51 minikube kubelet[3171]: E0110 04:44:51.903058    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.003244    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.103429    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.164525    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.165022    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.169157    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.204221    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.304554    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: I0110 04:44:52.347051    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:52 minikube kubelet[3171]: I0110 04:44:52.349577    3171 kubelet_node_status.go:70] Attempting to register node minikube
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.349969    3171 kubelet_node_status.go:92] Unable to register node "minikube" with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.404986    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.505094    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: I0110 04:44:52.554400    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:44:52 minikube kubelet[3171]: W0110 04:44:52.555941    3171 pod_container_deletor.go:75] Container "6660fc8036cfd792e5998de838cb75f5d0e78548aae4bb7d9d040631503323f3" not found in pod's containers
Jan 10 04:44:52 minikube kubelet[3171]: W0110 04:44:52.561531    3171 pod_container_deletor.go:75] Container "4d05bbeda408cef2bc1db1a3dae23730cae7b43f6e5779f87bb687420f247c68" not found in pod's containers
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.605265    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.705451    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.805856    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:52 minikube kubelet[3171]: E0110 04:44:52.906389    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.006560    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.106824    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.165161    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.166464    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.169626    3171 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.207412    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.307666    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.408018    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.508363    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.590073    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "fe1a6d1934a651ca3e40092710098882e2bb6f56c91f389f40364c03fa16886a" for pod "kube-addon-manager-minikube_kube-system(d682efea6fd7d1c11b13f78e8c81af08)" error: rpc error: code = Unknown desc = Error: No such container: fe1a6d1934a651ca3e40092710098882e2bb6f56c91f389f40364c03fa16886a
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.590096    3171 generic.go:241] PLEG: Ignoring events for pod kube-addon-manager-minikube/kube-system: rpc error: code = Unknown desc = Error: No such container: fe1a6d1934a651ca3e40092710098882e2bb6f56c91f389f40364c03fa16886a
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.609217    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.709444    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.809745    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:53 minikube kubelet[3171]: E0110 04:44:53.910016    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:54 minikube kubelet[3171]: E0110 04:44:54.010166    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:44:54 minikube kubelet[3171]: E0110 04:44:54.110385    3171 kubelet.go:2236] node "minikube" not found
...
...
...
Jan 10 04:45:38 minikube kubelet[3171]: E0110 04:45:38.102884    3171 kubelet.go:2236] node "minikube" not found
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.187064    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.187444    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.187482    3171 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Jan 10 04:45:38 minikube kubelet[3171]: E0110 04:45:38.220985    3171 reflector.go:134] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:minikube" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
Jan 10 04:45:38 minikube kubelet[3171]: E0110 04:45:38.221225    3171 reflector.go:134] object-"kube-system"/"coredns-token-wtfp2": Failed to list *v1.Secret: secrets "coredns-token-wtfp2" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system"
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.307337    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-wtfp2" (UniqueName: "kubernetes.io/secret/120104a5-0995-11e9-8587-080027afed9b-coredns-token-wtfp2") pod "coredns-576cbf47c7-cgmgz" (UID: "120104a5-0995-11e9-8587-080027afed9b")
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.307695    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/120104a5-0995-11e9-8587-080027afed9b-config-volume") pod "coredns-576cbf47c7-cgmgz" (UID: "120104a5-0995-11e9-8587-080027afed9b")
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.407927    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-wtfp2" (UniqueName: "kubernetes.io/secret/11856e6d-0995-11e9-8587-080027afed9b-coredns-token-wtfp2") pod "coredns-576cbf47c7-rqdt7" (UID: "11856e6d-0995-11e9-8587-080027afed9b")
Jan 10 04:45:38 minikube kubelet[3171]: I0110 04:45:38.408035    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/11856e6d-0995-11e9-8587-080027afed9b-config-volume") pod "coredns-576cbf47c7-rqdt7" (UID: "11856e6d-0995-11e9-8587-080027afed9b")
Jan 10 04:45:39 minikube kubelet[3171]: I0110 04:45:39.111812    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-tqr5w" (UniqueName: "kubernetes.io/secret/17742f43-0995-11e9-8587-080027afed9b-default-token-tqr5w") pod "kubernetes-dashboard-5bff5f8fb8-w7gj7" (UID: "17742f43-0995-11e9-8587-080027afed9b")
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.408243    3171 configmap.go:195] Couldn't get configMap kube-system/coredns: couldn't propagate object cache: timed out waiting for the condition
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.408945    3171 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/configmap/120104a5-0995-11e9-8587-080027afed9b-config-volume\" (\"120104a5-0995-11e9-8587-080027afed9b\")" failed. No retries permitted until 2019-01-10 04:45:39.908908395 +0000 UTC m=+50.838642678 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/120104a5-0995-11e9-8587-080027afed9b-config-volume\") pod \"coredns-576cbf47c7-cgmgz\" (UID: \"120104a5-0995-11e9-8587-080027afed9b\") : couldn't propagate object cache: timed out waiting for the condition"
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.409972    3171 secret.go:194] Couldn't get secret kube-system/coredns-token-wtfp2: couldn't propagate object cache: timed out waiting for the condition
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.410046    3171 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/secret/120104a5-0995-11e9-8587-080027afed9b-coredns-token-wtfp2\" (\"120104a5-0995-11e9-8587-080027afed9b\")" failed. No retries permitted until 2019-01-10 04:45:39.910017818 +0000 UTC m=+50.839752085 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"coredns-token-wtfp2\" (UniqueName: \"kubernetes.io/secret/120104a5-0995-11e9-8587-080027afed9b-coredns-token-wtfp2\") pod \"coredns-576cbf47c7-cgmgz\" (UID: \"120104a5-0995-11e9-8587-080027afed9b\") : couldn't propagate object cache: timed out waiting for the condition"
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.509957    3171 configmap.go:195] Couldn't get configMap kube-system/coredns: couldn't propagate object cache: timed out waiting for the condition
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.510109    3171 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/configmap/11856e6d-0995-11e9-8587-080027afed9b-config-volume\" (\"11856e6d-0995-11e9-8587-080027afed9b\")" failed. No retries permitted until 2019-01-10 04:45:40.01008874 +0000 UTC m=+50.939822996 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11856e6d-0995-11e9-8587-080027afed9b-config-volume\") pod \"coredns-576cbf47c7-rqdt7\" (UID: \"11856e6d-0995-11e9-8587-080027afed9b\") : couldn't propagate object cache: timed out waiting for the condition"
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.509980    3171 secret.go:194] Couldn't get secret kube-system/coredns-token-wtfp2: couldn't propagate object cache: timed out waiting for the condition
Jan 10 04:45:39 minikube kubelet[3171]: E0110 04:45:39.510915    3171 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/secret/11856e6d-0995-11e9-8587-080027afed9b-coredns-token-wtfp2\" (\"11856e6d-0995-11e9-8587-080027afed9b\")" failed. No retries permitted until 2019-01-10 04:45:40.010895585 +0000 UTC m=+50.940629839 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"coredns-token-wtfp2\" (UniqueName: \"kubernetes.io/secret/11856e6d-0995-11e9-8587-080027afed9b-coredns-token-wtfp2\") pod \"coredns-576cbf47c7-rqdt7\" (UID: \"11856e6d-0995-11e9-8587-080027afed9b\") : couldn't propagate object cache: timed out waiting for the condition"
Jan 10 04:45:39 minikube kubelet[3171]: I0110 04:45:39.614015    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-kjqzk" (UniqueName: "kubernetes.io/secret/1b51a550-0995-11e9-8587-080027afed9b-storage-provisioner-token-kjqzk") pod "storage-provisioner" (UID: "1b51a550-0995-11e9-8587-080027afed9b")
Jan 10 04:45:39 minikube kubelet[3171]: I0110 04:45:39.614108    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/1b51a550-0995-11e9-8587-080027afed9b-tmp") pod "storage-provisioner" (UID: "1b51a550-0995-11e9-8587-080027afed9b")
Jan 10 04:45:39 minikube kubelet[3171]: I0110 04:45:39.916150    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-tqr5w" (UniqueName: "kubernetes.io/secret/c52588f1-0995-11e9-aeb6-080027afed9b-default-token-tqr5w") pod "default-http-backend-5957bfbccb-ck77n" (UID: "c52588f1-0995-11e9-aeb6-080027afed9b")
Jan 10 04:45:40 minikube kubelet[3171]: W0110 04:45:40.277402    3171 container.go:393] Failed to create summary reader for "/system.slice/run-rc0155089d68545459c7060847de44c25.scope": none of the resources are being tracked.
Jan 10 04:45:40 minikube kubelet[3171]: W0110 04:45:40.278085    3171 container.go:393] Failed to create summary reader for "/system.slice/run-r9eeab794475845e7b7af912cf8010ac9.scope": none of the resources are being tracked.
Jan 10 04:45:40 minikube kubelet[3171]: I0110 04:45:40.318499    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "nginx-ingress-token-pg4kp" (UniqueName: "kubernetes.io/secret/c52641e2-0995-11e9-aeb6-080027afed9b-nginx-ingress-token-pg4kp") pod "nginx-ingress-controller-6958898f8f-t8m7z" (UID: "c52641e2-0995-11e9-aeb6-080027afed9b")
Jan 10 04:45:40 minikube kubelet[3171]: E0110 04:45:40.396690    3171 helpers.go:135] readString: Failed to read "/sys/fs/cgroup/cpu,cpuacct/system.slice/run-ra5390da237e14273b66ae7773b129779.scope/cpu.cfs_quota_us": read /sys/fs/cgroup/cpu,cpuacct/system.slice/run-ra5390da237e14273b66ae7773b129779.scope/cpu.cfs_quota_us: no such device
Jan 10 04:45:40 minikube kubelet[3171]: I0110 04:45:40.720784    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tiller-token-b2r48" (UniqueName: "kubernetes.io/secret/ecb79a8c-0999-11e9-a760-080027afed9b-tiller-token-b2r48") pod "tiller-deploy-7f94669995-vmkmq" (UID: "ecb79a8c-0999-11e9-a760-080027afed9b")
Jan 10 04:45:41 minikube kubelet[3171]: E0110 04:45:41.822084    3171 secret.go:194] Couldn't get secret kube-system/tiller-token-b2r48: couldn't propagate object cache: timed out waiting for the condition
Jan 10 04:45:41 minikube kubelet[3171]: E0110 04:45:41.822308    3171 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/secret/ecb79a8c-0999-11e9-a760-080027afed9b-tiller-token-b2r48\" (\"ecb79a8c-0999-11e9-a760-080027afed9b\")" failed. No retries permitted until 2019-01-10 04:45:42.322247315 +0000 UTC m=+53.251981630 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"tiller-token-b2r48\" (UniqueName: \"kubernetes.io/secret/ecb79a8c-0999-11e9-a760-080027afed9b-tiller-token-b2r48\") pod \"tiller-deploy-7f94669995-vmkmq\" (UID: \"ecb79a8c-0999-11e9-a760-080027afed9b\") : couldn't propagate object cache: timed out waiting for the condition"
Jan 10 04:45:41 minikube kubelet[3171]: I0110 04:45:41.926097    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/9aa6eae8-099a-11e9-a760-080027afed9b-config-volume") pod "rabbitmq-0" (UID: "9aa6eae8-099a-11e9-a760-080027afed9b")
Jan 10 04:45:41 minikube kubelet[3171]: I0110 04:45:41.926147    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "data" (UniqueName: "kubernetes.io/empty-dir/9aa6eae8-099a-11e9-a760-080027afed9b-data") pod "rabbitmq-0" (UID: "9aa6eae8-099a-11e9-a760-080027afed9b")
Jan 10 04:45:41 minikube kubelet[3171]: I0110 04:45:41.926167    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "rabbitmq-token-qh6rp" (UniqueName: "kubernetes.io/secret/9aa6eae8-099a-11e9-a760-080027afed9b-rabbitmq-token-qh6rp") pod "rabbitmq-0" (UID: "9aa6eae8-099a-11e9-a760-080027afed9b")
Jan 10 04:45:42 minikube kubelet[3171]: W0110 04:45:42.325563    3171 container.go:393] Failed to create summary reader for "/system.slice/run-re466bdacf7f541e981da19b4e539af2c.scope": none of the resources are being tracked.
Jan 10 04:45:43 minikube kubelet[3171]: I0110 04:45:43.548135    3171 kubelet_node_status.go:112] Node minikube was previously registered
Jan 10 04:45:43 minikube kubelet[3171]: I0110 04:45:43.548162    3171 kubelet_node_status.go:73] Successfully registered node minikube
Jan 10 04:45:43 minikube kubelet[3171]: I0110 04:45:43.639808    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-275ee215-099b-11e9-a760-080027afed9b" (UniqueName: "kubernetes.io/host-path/282d0d6a-099b-11e9-a760-080027afed9b-pvc-275ee215-099b-11e9-a760-080027afed9b") pod "mongo-mongodb-7d4b959d48-szjdp" (UID: "282d0d6a-099b-11e9-a760-080027afed9b")
Jan 10 04:45:43 minikube kubelet[3171]: I0110 04:45:43.640154    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-6gzp6" (UniqueName: "kubernetes.io/secret/282d0d6a-099b-11e9-a760-080027afed9b-default-token-6gzp6") pod "mongo-mongodb-7d4b959d48-szjdp" (UID: "282d0d6a-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: E0110 04:45:44.337250    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "5c592f84a3cefc7c5631ecab2f150bceedfad63f222b87c651582954b13bf58c" for pod "storage-provisioner_kube-system(1b51a550-0995-11e9-8587-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 5c592f84a3cefc7c5631ecab2f150bceedfad63f222b87c651582954b13bf58c
Jan 10 04:45:44 minikube kubelet[3171]: E0110 04:45:44.337367    3171 generic.go:241] PLEG: Ignoring events for pod storage-provisioner/kube-system: rpc error: code = Unknown desc = Error: No such container: 5c592f84a3cefc7c5631ecab2f150bceedfad63f222b87c651582954b13bf58c
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744454    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-6gzp6" (UniqueName: "kubernetes.io/secret/659eeea9-099b-11e9-a760-080027afed9b-default-token-6gzp6") pod "redis-slave-5d645bb654-dgk6c" (UID: "659eeea9-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744516    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-659ee1b4-099b-11e9-a760-080027afed9b" (UniqueName: "kubernetes.io/host-path/659f77d1-099b-11e9-a760-080027afed9b-pvc-659ee1b4-099b-11e9-a760-080027afed9b") pod "redis-master-0" (UID: "659f77d1-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744545    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "health" (UniqueName: "kubernetes.io/configmap/659f77d1-099b-11e9-a760-080027afed9b-health") pod "redis-master-0" (UID: "659f77d1-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744566    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "health" (UniqueName: "kubernetes.io/configmap/659eeea9-099b-11e9-a760-080027afed9b-health") pod "redis-slave-5d645bb654-dgk6c" (UID: "659eeea9-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744586    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config" (UniqueName: "kubernetes.io/configmap/659f77d1-099b-11e9-a760-080027afed9b-config") pod "redis-master-0" (UID: "659f77d1-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744604    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-6gzp6" (UniqueName: "kubernetes.io/secret/659f77d1-099b-11e9-a760-080027afed9b-default-token-6gzp6") pod "redis-master-0" (UID: "659f77d1-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744626    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config" (UniqueName: "kubernetes.io/configmap/659eeea9-099b-11e9-a760-080027afed9b-config") pod "redis-slave-5d645bb654-dgk6c" (UID: "659eeea9-099b-11e9-a760-080027afed9b")
Jan 10 04:45:44 minikube kubelet[3171]: I0110 04:45:44.744648    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "redis-data" (UniqueName: "kubernetes.io/empty-dir/659eeea9-099b-11e9-a760-080027afed9b-redis-data") pod "redis-slave-5d645bb654-dgk6c" (UID: "659eeea9-099b-11e9-a760-080027afed9b")
Jan 10 04:45:45 minikube kubelet[3171]: I0110 04:45:45.550266    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-6gzp6" (UniqueName: "kubernetes.io/secret/a8273be4-09a4-11e9-a760-080027afed9b-default-token-6gzp6") pod "mysql-f69fbb8fd-r56pj" (UID: "a8273be4-09a4-11e9-a760-080027afed9b")
Jan 10 04:45:45 minikube kubelet[3171]: I0110 04:45:45.550375    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-a73e672a-09a4-11e9-a760-080027afed9b" (UniqueName: "kubernetes.io/host-path/a8273be4-09a4-11e9-a760-080027afed9b-pvc-a73e672a-09a4-11e9-a760-080027afed9b") pod "mysql-f69fbb8fd-r56pj" (UID: "a8273be4-09a4-11e9-a760-080027afed9b")
Jan 10 04:45:46 minikube kubelet[3171]: E0110 04:45:46.217222    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f" for pod "coredns-576cbf47c7-cgmgz_kube-system(120104a5-0995-11e9-8587-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f
Jan 10 04:45:46 minikube kubelet[3171]: E0110 04:45:46.217672    3171 generic.go:241] PLEG: Ignoring events for pod coredns-576cbf47c7-cgmgz/kube-system: rpc error: code = Unknown desc = Error: No such container: bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f
Jan 10 04:45:47 minikube kubelet[3171]: E0110 04:45:47.235982    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "8350bd984a1874c9b967f47575ae8ffef3594ce208faf2bf12e7c39feb91c31e" for pod "coredns-576cbf47c7-rqdt7_kube-system(11856e6d-0995-11e9-8587-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 8350bd984a1874c9b967f47575ae8ffef3594ce208faf2bf12e7c39feb91c31e
Jan 10 04:45:47 minikube kubelet[3171]: E0110 04:45:47.236307    3171 generic.go:241] PLEG: Ignoring events for pod coredns-576cbf47c7-rqdt7/kube-system: rpc error: code = Unknown desc = Error: No such container: 8350bd984a1874c9b967f47575ae8ffef3594ce208faf2bf12e7c39feb91c31e
Jan 10 04:45:47 minikube kubelet[3171]: E0110 04:45:47.246577    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f" for pod "coredns-576cbf47c7-cgmgz_kube-system(120104a5-0995-11e9-8587-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f
Jan 10 04:45:47 minikube kubelet[3171]: E0110 04:45:47.246673    3171 generic.go:241] PLEG: Ignoring events for pod coredns-576cbf47c7-cgmgz/kube-system: rpc error: code = Unknown desc = Error: No such container: bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f
Jan 10 04:45:47 minikube kubelet[3171]: E0110 04:45:47.248551    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f" for pod "coredns-576cbf47c7-cgmgz_kube-system(120104a5-0995-11e9-8587-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f
Jan 10 04:45:47 minikube kubelet[3171]: E0110 04:45:47.248627    3171 generic.go:271] PLEG: pod coredns-576cbf47c7-cgmgz/kube-system failed reinspection: rpc error: code = Unknown desc = Error: No such container: bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f
Jan 10 04:45:49 minikube kubelet[3171]: W0110 04:45:49.726142    3171 docker_sandbox.go:375] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-576cbf47c7-rqdt7 through plugin: invalid network status for
Jan 10 04:45:49 minikube kubelet[3171]: I0110 04:45:49.969116    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-6gzp6" (UniqueName: "kubernetes.io/secret/b3e6f8a3-131e-11e9-8151-080027afed9b-default-token-6gzp6") pod "minio-dd7466588-bkxb5" (UID: "b3e6f8a3-131e-11e9-8151-080027afed9b")
Jan 10 04:45:49 minikube kubelet[3171]: I0110 04:45:49.969183    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-b2829dc0-131e-11e9-8151-080027afed9b" (UniqueName: "kubernetes.io/host-path/b3e6f8a3-131e-11e9-8151-080027afed9b-pvc-b2829dc0-131e-11e9-8151-080027afed9b") pod "minio-dd7466588-bkxb5" (UID: "b3e6f8a3-131e-11e9-8151-080027afed9b")
Jan 10 04:45:49 minikube kubelet[3171]: I0110 04:45:49.969217    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "micro-services-token-lf9nn" (UniqueName: "kubernetes.io/secret/b7f223e5-1409-11e9-ba56-080027afed9b-micro-services-token-lf9nn") pod "micro-7bbd8cb5fd-xlq2f" (UID: "b7f223e5-1409-11e9-ba56-080027afed9b")
Jan 10 04:45:49 minikube kubelet[3171]: I0110 04:45:49.969241    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "minio-server-config" (UniqueName: "kubernetes.io/configmap/b3e6f8a3-131e-11e9-8151-080027afed9b-minio-server-config") pod "minio-dd7466588-bkxb5" (UID: "b3e6f8a3-131e-11e9-8151-080027afed9b")
Jan 10 04:45:49 minikube kubelet[3171]: I0110 04:45:49.969264    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "minio-user" (UniqueName: "kubernetes.io/secret/b3e6f8a3-131e-11e9-8151-080027afed9b-minio-user") pod "minio-dd7466588-bkxb5" (UID: "b3e6f8a3-131e-11e9-8151-080027afed9b")
Jan 10 04:45:49 minikube kubelet[3171]: I0110 04:45:49.969291    3171 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "minio-config-dir" (UniqueName: "kubernetes.io/empty-dir/b3e6f8a3-131e-11e9-8151-080027afed9b-minio-config-dir") pod "minio-dd7466588-bkxb5" (UID: "b3e6f8a3-131e-11e9-8151-080027afed9b")
Jan 10 04:45:50 minikube kubelet[3171]: W0110 04:45:50.200769    3171 container.go:393] Failed to create summary reader for "/system.slice/run-r32dba1cd56eb45809e44cba8cf30ef5d.scope": none of the resources are being tracked.
Jan 10 04:45:50 minikube kubelet[3171]: E0110 04:45:50.755353    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f" for pod "default-http-backend-5957bfbccb-ck77n_kube-system(c52588f1-0995-11e9-aeb6-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f
Jan 10 04:45:50 minikube kubelet[3171]: E0110 04:45:50.755851    3171 generic.go:241] PLEG: Ignoring events for pod default-http-backend-5957bfbccb-ck77n/kube-system: rpc error: code = Unknown desc = Error: No such container: 654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f
Jan 10 04:45:51 minikube kubelet[3171]: E0110 04:45:51.783038    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "72084daa83db1798c56b8cf31f071ecdc29e4ecf86499b95e560299fd2bfb18b" for pod "nginx-ingress-controller-6958898f8f-t8m7z_kube-system(c52641e2-0995-11e9-aeb6-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 72084daa83db1798c56b8cf31f071ecdc29e4ecf86499b95e560299fd2bfb18b
Jan 10 04:45:51 minikube kubelet[3171]: E0110 04:45:51.785624    3171 generic.go:241] PLEG: Ignoring events for pod nginx-ingress-controller-6958898f8f-t8m7z/kube-system: rpc error: code = Unknown desc = Error: No such container: 72084daa83db1798c56b8cf31f071ecdc29e4ecf86499b95e560299fd2bfb18b
Jan 10 04:45:51 minikube kubelet[3171]: E0110 04:45:51.787774    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f" for pod "default-http-backend-5957bfbccb-ck77n_kube-system(c52588f1-0995-11e9-aeb6-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f
Jan 10 04:45:51 minikube kubelet[3171]: E0110 04:45:51.787810    3171 generic.go:241] PLEG: Ignoring events for pod default-http-backend-5957bfbccb-ck77n/kube-system: rpc error: code = Unknown desc = Error: No such container: 654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f
Jan 10 04:45:51 minikube kubelet[3171]: E0110 04:45:51.789443    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f" for pod "default-http-backend-5957bfbccb-ck77n_kube-system(c52588f1-0995-11e9-aeb6-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f
Jan 10 04:45:51 minikube kubelet[3171]: E0110 04:45:51.789470    3171 generic.go:271] PLEG: pod default-http-backend-5957bfbccb-ck77n/kube-system failed reinspection: rpc error: code = Unknown desc = Error: No such container: 654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f
Jan 10 04:45:52 minikube kubelet[3171]: W0110 04:45:52.969834    3171 pod_container_deletor.go:75] Container "6744419559b89aa7429c16c71e960284c1c079e9d8a6affee2102306fd18b7c3" not found in pod's containers
Jan 10 04:45:52 minikube kubelet[3171]: W0110 04:45:52.969891    3171 pod_container_deletor.go:75] Container "5c592f84a3cefc7c5631ecab2f150bceedfad63f222b87c651582954b13bf58c" not found in pod's containers
Jan 10 04:45:52 minikube kubelet[3171]: W0110 04:45:52.973117    3171 kubelet.go:1622] Deleting mirror pod "kube-controller-manager-minikube_kube-system(a8b836cc-1491-11e9-8f29-080027afed9b)" because it is outdated
Jan 10 04:45:52 minikube kubelet[3171]: I0110 04:45:52.994982    3171 reconciler.go:154] Reconciler: start to sync state
Jan 10 04:45:53 minikube kubelet[3171]: W0110 04:45:53.475552    3171 pod_container_deletor.go:75] Container "bc5fd7d0b35333691dad7a586ea2bf5a396cc8ed177e1d7995cb8a1eaa8c0e2f" not found in pod's containers
Jan 10 04:45:53 minikube kubelet[3171]: W0110 04:45:53.475615    3171 pod_container_deletor.go:75] Container "8350bd984a1874c9b967f47575ae8ffef3594ce208faf2bf12e7c39feb91c31e" not found in pod's containers
Jan 10 04:45:54 minikube kubelet[3171]: W0110 04:45:54.371001    3171 docker_sandbox.go:375] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/default-http-backend-5957bfbccb-ck77n through plugin: invalid network status for
Jan 10 04:45:54 minikube kubelet[3171]: W0110 04:45:54.386003    3171 pod_container_deletor.go:75] Container "654537be80e144b9c773c480f15abb7c7c171f21b7be3f6106a2853d8bda9d9f" not found in pod's containers
Jan 10 04:45:54 minikube kubelet[3171]: W0110 04:45:54.757540    3171 pod_container_deletor.go:75] Container "72084daa83db1798c56b8cf31f071ecdc29e4ecf86499b95e560299fd2bfb18b" not found in pod's containers
Jan 10 04:45:55 minikube kubelet[3171]: E0110 04:45:55.777930    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289" for pod "rabbitmq-0_default(9aa6eae8-099a-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289
Jan 10 04:45:55 minikube kubelet[3171]: E0110 04:45:55.778026    3171 generic.go:241] PLEG: Ignoring events for pod rabbitmq-0/default: rpc error: code = Unknown desc = Error: No such container: e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289
Jan 10 04:45:55 minikube kubelet[3171]: E0110 04:45:55.779503    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72" for pod "tiller-deploy-7f94669995-vmkmq_kube-system(ecb79a8c-0999-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:55 minikube kubelet[3171]: E0110 04:45:55.779533    3171 generic.go:241] PLEG: Ignoring events for pod tiller-deploy-7f94669995-vmkmq/kube-system: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.792371    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72" for pod "tiller-deploy-7f94669995-vmkmq_kube-system(ecb79a8c-0999-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.792722    3171 generic.go:241] PLEG: Ignoring events for pod tiller-deploy-7f94669995-vmkmq/kube-system: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.800437    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289" for pod "rabbitmq-0_default(9aa6eae8-099a-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.800492    3171 generic.go:241] PLEG: Ignoring events for pod rabbitmq-0/default: rpc error: code = Unknown desc = Error: No such container: e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.802104    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289" for pod "rabbitmq-0_default(9aa6eae8-099a-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.802133    3171 generic.go:271] PLEG: pod rabbitmq-0/default failed reinspection: rpc error: code = Unknown desc = Error: No such container: e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.803760    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72" for pod "tiller-deploy-7f94669995-vmkmq_kube-system(ecb79a8c-0999-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:56 minikube kubelet[3171]: E0110 04:45:56.803801    3171 generic.go:271] PLEG: pod tiller-deploy-7f94669995-vmkmq/kube-system failed reinspection: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:57 minikube kubelet[3171]: W0110 04:45:57.835111    3171 pod_container_deletor.go:75] Container "e92ddf66a2c7be2fba697f2773fdc27185ca12e36f4554f32ec0352110292289" not found in pod's containers
Jan 10 04:45:57 minikube kubelet[3171]: E0110 04:45:57.842789    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72" for pod "tiller-deploy-7f94669995-vmkmq_kube-system(ecb79a8c-0999-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:57 minikube kubelet[3171]: E0110 04:45:57.842886    3171 generic.go:241] PLEG: Ignoring events for pod tiller-deploy-7f94669995-vmkmq/kube-system: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:57 minikube kubelet[3171]: E0110 04:45:57.849840    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72" for pod "tiller-deploy-7f94669995-vmkmq_kube-system(ecb79a8c-0999-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:45:57 minikube kubelet[3171]: E0110 04:45:57.849870    3171 generic.go:271] PLEG: pod tiller-deploy-7f94669995-vmkmq/kube-system failed reinspection: rpc error: code = Unknown desc = Error: No such container: 33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72
Jan 10 04:46:00 minikube kubelet[3171]: W0110 04:46:00.212075    3171 pod_container_deletor.go:75] Container "33690e2ea35897ee7cbf1c0831a28fc7339a1562c25620a23d72e06815ed1a72" not found in pod's containers
Jan 10 04:46:00 minikube kubelet[3171]: E0110 04:46:00.215183    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0" for pod "mongo-mongodb-7d4b959d48-szjdp_default(282d0d6a-099b-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0
Jan 10 04:46:00 minikube kubelet[3171]: E0110 04:46:00.215215    3171 generic.go:241] PLEG: Ignoring events for pod mongo-mongodb-7d4b959d48-szjdp/default: rpc error: code = Unknown desc = Error: No such container: af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0
Jan 10 04:46:00 minikube kubelet[3171]: W0110 04:46:00.640597    3171 kuberuntime_container.go:683] No ref for container {"docker" "41125ba23bad9a2e6bb724a7a9a28a708e076e135fab5cb6a5305c918d0c1962"}
Jan 10 04:46:01 minikube kubelet[3171]: E0110 04:46:01.235041    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0" for pod "mongo-mongodb-7d4b959d48-szjdp_default(282d0d6a-099b-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0
Jan 10 04:46:01 minikube kubelet[3171]: E0110 04:46:01.235080    3171 generic.go:241] PLEG: Ignoring events for pod mongo-mongodb-7d4b959d48-szjdp/default: rpc error: code = Unknown desc = Error: No such container: af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0
Jan 10 04:46:01 minikube kubelet[3171]: W0110 04:46:01.243407    3171 pod_container_deletor.go:75] Container "41125ba23bad9a2e6bb724a7a9a28a708e076e135fab5cb6a5305c918d0c1962" not found in pod's containers
Jan 10 04:46:01 minikube kubelet[3171]: E0110 04:46:01.245319    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "12f41fc5a8e673d5c00244d131ffb4bc5271d1cab0e9912ec7159e9c95976e22" for pod "redis-slave-5d645bb654-dgk6c_default(659eeea9-099b-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: 12f41fc5a8e673d5c00244d131ffb4bc5271d1cab0e9912ec7159e9c95976e22
Jan 10 04:46:01 minikube kubelet[3171]: E0110 04:46:01.245365    3171 generic.go:241] PLEG: Ignoring events for pod redis-slave-5d645bb654-dgk6c/default: rpc error: code = Unknown desc = Error: No such container: 12f41fc5a8e673d5c00244d131ffb4bc5271d1cab0e9912ec7159e9c95976e22
Jan 10 04:46:01 minikube kubelet[3171]: E0110 04:46:01.247035    3171 kuberuntime_manager.go:852] PodSandboxStatus of sandbox "af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0" for pod "mongo-mongodb-7d4b959d48-szjdp_default(282d0d6a-099b-11e9-a760-080027afed9b)" error: rpc error: code = Unknown desc = Error: No such container: af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0
Jan 10 04:46:01 minikube kubelet[3171]: E0110 04:46:01.247120    3171 generic.go:271] PLEG: pod mongo-mongodb-7d4b959d48-szjdp/default failed reinspection: rpc error: code = Unknown desc = Error: No such container: af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0
Jan 10 04:46:02 minikube kubelet[3171]: W0110 04:46:02.783238    3171 pod_container_deletor.go:75] Container "af93efb07ecc0ca79b50a81a2420fbce87fa03eee547b00416b2154852b246c0" not found in pod's containers
Jan 10 04:46:03 minikube kubelet[3171]: W0110 04:46:03.275363    3171 kuberuntime_container.go:683] No ref for container {"docker" "e0afdd37407430aa9f4234e872a3034491b3c4bbf8dab09ba66371cbe78735a0"}
Jan 10 04:46:27 minikube kubelet[3171]: W0110 04:46:27.660312    3171 pod_container_deletor.go:75] Container "12f41fc5a8e673d5c00244d131ffb4bc5271d1cab0e9912ec7159e9c95976e22" not found in pod's containers
Jan 10 04:46:28 minikube kubelet[3171]: W0110 04:46:28.728448    3171 container.go:393] Failed to create summary reader for "/system.slice/run-ra370c12b57e944de8736116160fdea16.scope": none of the resources are being tracked.
Jan 10 04:46:28 minikube kubelet[3171]: W0110 04:46:28.729774    3171 container.go:393] Failed to create summary reader for "/system.slice/run-r6d85dfa99f114a10ad07c08b235242a1.scope": none of the resources are being tracked.
Jan 10 04:46:32 minikube kubelet[3171]: W0110 04:46:32.676660    3171 kubelet.go:1622] Deleting mirror pod "kube-controller-manager-minikube_kube-system(a8b836cc-1491-11e9-8f29-080027afed9b)" because it is outdated
Jan 10 04:46:32 minikube kubelet[3171]: W0110 04:46:32.698369    3171 pod_container_deletor.go:75] Container "e0afdd37407430aa9f4234e872a3034491b3c4bbf8dab09ba66371cbe78735a0" not found in pod's containers
Jan 10 04:46:32 minikube kubelet[3171]: W0110 04:46:32.698436    3171 pod_container_deletor.go:75] Container "6edaad87cf1eb787d62b1503d18be7211ee1e296ec19eb02f56280ccdd0048fa" not found in pod's containers
Jan 10 04:46:43 minikube kubelet[3171]: E0110 04:46:43.852110    3171 remote_runtime.go:282] ContainerStatus "5d1719583551b4e11c14072b8f9cc0e18d0fc4881c9c37c777d888c8ec9353e9" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 5d1719583551b4e11c14072b8f9cc0e18d0fc4881c9c37c777d888c8ec9353e9
Jan 10 04:46:43 minikube kubelet[3171]: E0110 04:46:43.852155    3171 kuberuntime_container.go:393] ContainerStatus for 5d1719583551b4e11c14072b8f9cc0e18d0fc4881c9c37c777d888c8ec9353e9 error: rpc error: code = Unknown desc = Error: No such container: 5d1719583551b4e11c14072b8f9cc0e18d0fc4881c9c37c777d888c8ec9353e9
Jan 10 04:46:43 minikube kubelet[3171]: E0110 04:46:43.852163    3171 kuberuntime_manager.go:866] getPodContainerStatuses for pod "minio-dd7466588-bkxb5_default(b3e6f8a3-131e-11e9-8151-080027afed9b)" failed: rpc error: code = Unknown desc = Error: No such container: 5d1719583551b4e11c14072b8f9cc0e18d0fc4881c9c37c777d888c8ec9353e9
Jan 10 04:46:43 minikube kubelet[3171]: E0110 04:46:43.852174    3171 generic.go:241] PLEG: Ignoring events for pod minio-dd7466588-bkxb5/default: rpc error: code = Unknown desc = Error: No such container: 5d1719583551b4e11c14072b8f9cc0e18d0fc4881c9c37c777d888c8ec9353e9

Anything else do we need to know:

@erkolson
Copy link

Same but on OS X 10.13.6

@tstromberg
Copy link
Contributor

Do you mind trying again with minikube v0.33.0? We've made a few changes which might help here.

If it still fails, can you attach the apiserver logs? I suspect it may be crash-looping:

minikube ssh 'docker logs $(docker ps -a --filter name=k8s_kube-apiserver --format={{.ID}})'

@prksu
Copy link
Author

prksu commented Jan 19, 2019

@tstromberg look good
I will try it and will report it later. thanks

@prksu
Copy link
Author

prksu commented Jan 20, 2019

@tstromberg I tried it again with minikube v0.33.0 but on my MacBook (OS X 10.13.6) with vm driver: virtualbox. it starts properly!

$ minikube stop
Stopping local Kubernetes cluster...
Machine stopped.
$ minikube start
Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Machine exists, restarting cluster components...
Verifying kubelet health ...
Verifying apiserver health ....
Kubectl is now configured to use the cluster.
Loading cached images from config file.


Everything looks great. Please enjoy minikube!
$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

@prksu prksu closed this as completed Jan 20, 2019
@tomfloresa
Copy link

I'm still getting the error. I'm running minikube v0.33.1 in macOs 10.13.6. It boots up normally but after creation and attempting to make some deploys, it crushes and stops working.

@aaronkhawkins
Copy link

I'm getting the same error - everything looks fine but then the apiserver fails - the dashboard freezes, commands don't work. I'm on v0.33.1 in Mac 10.14.2

@adubkov
Copy link

adubkov commented Feb 7, 2019

I can confirm on 10.13.6 apiserver shows error and everything stop working some time after deployment.

@adubkov
Copy link

adubkov commented Feb 8, 2019

Not sure if that will help. Logs from k8s_kube-apiserver for the time it shows error.

Trace[357906370]: [2.787307031s] [2.787302411s] About to write a response
I0208 03:20:38.947702       1 trace.go:76] Trace[188527786]: "Get /api/v1/namespaces/local-chain" (started: 2019-02-08 03:20:36.211232175 +0000 UTC m=+12341.920222245) (total time: 2.736442306s):
Trace[188527786]: [2.734829643s] [2.734622281s] About to write a response
I0208 03:20:38.978480       1 trace.go:76] Trace[1223047074]: "Get /api/v1/namespaces/kube-system/services/kubernetes-dashboard" (started: 2019-02-08 03:20:38.326474798 +0000 UTC m=+12344.035464907) (total time: 651.929614ms):
Trace[1223047074]: [651.753632ms] [651.489523ms] About to write a response
E0208 03:21:17.689694       1 upgradeaware.go:343] Error proxying data from client to backend: read tcp 192.168.99.100:8443->192.168.99.1:50333: read: connection reset by peer
E0208 03:22:45.772685       1 upgradeaware.go:343] Error proxying data from client to backend: read tcp 192.168.99.100:8443->192.168.99.1:50383: read: connection reset by peer
E0208 03:22:55.716975       1 upgradeaware.go:343] Error proxying data from client to backend: read tcp 192.168.99.100:8443->192.168.99.1:50407: read: connection reset by peer
E0208 03:22:55.841568       1 upgradeaware.go:343] Error proxying data from client to backend: read tcp 192.168.99.100:8443->192.168.99.1:50414: read: connection reset by peer
I0208 03:24:36.970211       1 trace.go:76] Trace[1556638121]: "GuaranteedUpdate etcd3: *core.Node" (started: 2019-02-08 03:24:36.414366583 +0000 UTC m=+12582.123356707) (total time: 549.187167ms):
Trace[1556638121]: [549.0897ms] [426.919259ms] Transaction committed
I0208 03:24:36.981210       1 trace.go:76] Trace[1954957489]: "Patch /api/v1/nodes/minikube/status" (started: 2019-02-08 03:24:36.394755772 +0000 UTC m=+12582.103745846) (total time: 586.427442ms):
Trace[1954957489]: [121.313298ms] [81.1604ms] About to check admission control
Trace[1954957489]: [586.026594ms] [464.713296ms] Object stored in database
I0208 03:25:26.392317       1 trace.go:76] Trace[1671364426]: "Get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:25:25.802079437 +0000 UTC m=+12631.511069555) (total time: 586.879318ms):
Trace[1671364426]: [585.286899ms] [583.744899ms] About to write a response
I0208 03:25:26.392374       1 trace.go:76] Trace[1320827465]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:25:25.861367468 +0000 UTC m=+12631.570357530) (total time: 527.649148ms):
Trace[1320827465]: [526.131979ms] [526.094073ms] About to write a response
I0208 03:25:27.338544       1 trace.go:76] Trace[596721458]: "Get /apis/apps/v1/namespaces/kube-system/deployments/kubernetes-dashboard" (started: 2019-02-08 03:25:26.734477944 +0000 UTC m=+12632.443468042) (total time: 596.653417ms):
Trace[596721458]: [581.567701ms] [581.561862ms] About to write a response
I0208 03:25:27.338992       1 trace.go:76] Trace[1502351283]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:25:26.713625563 +0000 UTC m=+12632.422615674) (total time: 617.129651ms):
Trace[1502351283]: [617.090857ms] [582.751514ms] Transaction committed
I0208 03:25:27.340369       1 trace.go:76] Trace[1884011824]: "Update /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:25:26.710548295 +0000 UTC m=+12632.419538364) (total time: 629.338354ms):
Trace[1884011824]: [628.683657ms] [627.404353ms] Object stored in database
I0208 03:25:27.370269       1 trace.go:76] Trace[168858714]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:25:26.75010833 +0000 UTC m=+12632.459098410) (total time: 620.122139ms):
Trace[168858714]: [619.519515ms] [619.152466ms] Transaction committed
I0208 03:25:27.370593       1 trace.go:76] Trace[459970387]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:25:26.706705271 +0000 UTC m=+12632.415695383) (total time: 663.873173ms):
Trace[459970387]: [663.762251ms] [653.974935ms] Object stored in database
I0208 03:25:55.991911       1 trace.go:76] Trace[1292792790]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:25:55.456905157 +0000 UTC m=+12661.165895220) (total time: 532.728943ms):
Trace[1292792790]: [532.09841ms] [504.043191ms] Object stored in database
I0208 03:26:05.108141       1 trace.go:76] Trace[58659429]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:26:04.462173894 +0000 UTC m=+12670.171164006) (total time: 555.498465ms):
Trace[58659429]: [117.970698ms] [117.970698ms] About to convert to expected version
Trace[58659429]: [554.831619ms] [436.802284ms] Object stored in database
I0208 03:26:08.261042       1 trace.go:76] Trace[829378175]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:26:07.631847348 +0000 UTC m=+12673.340837461) (total time: 626.867254ms):
Trace[829378175]: [626.662316ms] [619.882576ms] Transaction committed
I0208 03:26:08.262006       1 trace.go:76] Trace[2019053553]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:26:07.625683266 +0000 UTC m=+12673.334673334) (total time: 636.298619ms):
Trace[2019053553]: [635.922895ms] [633.109475ms] Object stored in database
I0208 03:26:08.336675       1 trace.go:76] Trace[324010794]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:26:07.792408406 +0000 UTC m=+12673.501398488) (total time: 544.232153ms):
Trace[324010794]: [544.007192ms] [543.967444ms] About to write a response
I0208 03:26:10.731421       1 trace.go:76] Trace[905884168]: "GuaranteedUpdate etcd3: *v1.Endpoints" (started: 2019-02-08 03:26:10.014830515 +0000 UTC m=+12675.723820596) (total time: 715.40459ms):
Trace[905884168]: [133.34107ms] [133.34107ms] initial value restored
Trace[905884168]: [441.088041ms] [307.746971ms] Transaction prepared
Trace[905884168]: [715.384919ms] [274.296878ms] Transaction committed
I0208 03:26:16.588182       1 trace.go:76] Trace[1352807689]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:26:15.82700875 +0000 UTC m=+12681.535998914) (total time: 672.573365ms):
Trace[1352807689]: [672.262337ms] [608.166808ms] Transaction committed
I0208 03:26:16.603633       1 trace.go:76] Trace[1015302004]: "Update /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:26:15.822092159 +0000 UTC m=+12681.531082239) (total time: 766.71849ms):
Trace[1015302004]: [766.352401ms] [764.24274ms] Object stored in database
I0208 03:26:16.704697       1 trace.go:76] Trace[817486945]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:26:16.098760085 +0000 UTC m=+12681.807750156) (total time: 605.897712ms):
Trace[817486945]: [605.122465ms] [559.852081ms] Object stored in database
I0208 03:26:22.222731       1 trace.go:76] Trace[317292325]: "GuaranteedUpdate etcd3: *v1.Endpoints" (started: 2019-02-08 03:26:21.417380811 +0000 UTC m=+12687.126370933) (total time: 738.959345ms):
Trace[317292325]: [435.540546ms] [431.280528ms] Transaction prepared
Trace[317292325]: [738.324215ms] [302.783669ms] Transaction committed
I0208 03:26:35.849823       1 trace.go:76] Trace[1351377241]: "Get /api/v1/nodes/minikube" (started: 2019-02-08 03:26:32.102688516 +0000 UTC m=+12697.811678648) (total time: 3.742070238s):
Trace[1351377241]: [639.590096ms] [630.836465ms] About to write a response
Trace[1351377241]: [3.741920892s] [3.089718733s] Transformed response object
E0208 03:26:36.031388       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}
I0208 03:26:36.661376       1 trace.go:76] Trace[1420542963]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:26:24.816559901 +0000 UTC m=+12690.525550078) (total time: 11.843471866s):
Trace[1420542963]: [11.843471866s] [11.841502085s] END
E0208 03:26:37.058958       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}
I0208 03:26:37.068665       1 trace.go:76] Trace[2038703541]: "Get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:26:24.825033906 +0000 UTC m=+12690.534023976) (total time: 12.243170838s):
Trace[2038703541]: [12.243170838s] [12.242769248s] END
I0208 03:26:38.947522       1 trace.go:76] Trace[1187907646]: "Get /api/v1/namespaces/kube-system/secrets/default-token-qg259" (started: 2019-02-08 03:26:38.103043525 +0000 UTC m=+12703.812033608) (total time: 842.036752ms):
Trace[1187907646]: [841.187375ms] [841.182959ms] About to write a response
I0208 03:26:39.473976       1 trace.go:76] Trace[340145448]: "GuaranteedUpdate etcd3: *core.Node" (started: 2019-02-08 03:26:37.176177519 +0000 UTC m=+12702.885167690) (total time: 2.297601695s):
Trace[340145448]: [341.792557ms] [335.780391ms] Transaction prepared
Trace[340145448]: [2.29681231s] [1.955019753s] Transaction committed
I0208 03:26:39.476195       1 trace.go:76] Trace[1183722355]: "Patch /api/v1/nodes/minikube/status" (started: 2019-02-08 03:26:37.051500864 +0000 UTC m=+12702.760490975) (total time: 2.424671455s):
Trace[1183722355]: [223.909146ms] [223.394311ms] About to apply patch
Trace[1183722355]: [2.423109497s] [2.167059745s] Object stored in database
I0208 03:26:39.509219       1 trace.go:76] Trace[1520765040]: "List /api/v1/namespaces/local-chain/pods" (started: 2019-02-08 03:26:37.134228811 +0000 UTC m=+12702.843218878) (total time: 2.374964836s):
Trace[1520765040]: [2.355071934s] [2.354870574s] Listing from storage done
I0208 03:26:39.576729       1 trace.go:76] Trace[842388082]: "Get /api/v1/namespaces/kube-system/secrets/cronjob-controller-token-p8dtq" (started: 2019-02-08 03:26:29.685064228 +0000 UTC m=+12695.394054422) (total time: 9.89142471s):
Trace[842388082]: [265.716894ms] [265.716894ms] About to Get from storage
Trace[842388082]: [9.890948653s] [9.625231759s] About to write a response
I0208 03:26:39.577247       1 trace.go:76] Trace[1623418527]: "Get /api/v1/namespaces/kube-system/secrets/pod-garbage-collector-token-t84ww" (started: 2019-02-08 03:26:37.043133308 +0000 UTC m=+12702.752123376) (total time: 2.533970289s):
Trace[1623418527]: [2.533240969s] [2.533238546s] About to write a response
I0208 03:26:39.578687       1 trace.go:76] Trace[1734802706]: "Get /api/v1/namespaces/default" (started: 2019-02-08 03:26:37.04308224 +0000 UTC m=+12702.752072341) (total time: 2.535552859s):
Trace[1734802706]: [2.534640127s] [2.534631796s] About to write a response
I0208 03:26:39.687056       1 trace.go:76] Trace[1892814940]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.rbac.authorization.k8s.io/status" (started: 2019-02-08 03:26:39.134474444 +0000 UTC m=+12704.843464513) (total time: 552.541761ms):
Trace[1892814940]: [117.620292ms] [117.620292ms] About to convert to expected version
Trace[1892814940]: [550.951418ms] [433.187591ms] Object stored in database
I0208 03:26:39.689004       1 trace.go:76] Trace[1933935133]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.apiextensions.k8s.io/status" (started: 2019-02-08 03:26:39.101296643 +0000 UTC m=+12704.810286715) (total time: 587.520151ms):
Trace[1933935133]: [587.092006ms] [553.202495ms] Object stored in database
I0208 03:26:39.694957       1 trace.go:76] Trace[1984593651]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.certificates.k8s.io/status" (started: 2019-02-08 03:26:39.101606314 +0000 UTC m=+12704.810596386) (total time: 591.106447ms):
Trace[1984593651]: [591.051114ms] [557.383157ms] Object stored in database
I0208 03:26:39.695848       1 trace.go:76] Trace[2013425737]: "Get /api/v1/namespaces/kube-system/pods/kube-controller-manager-minikube" (started: 2019-02-08 03:26:38.660393982 +0000 UTC m=+12704.369384049) (total time: 1.035031759s):
Trace[2013425737]: [1.034000018s] [1.033990618s] About to write a response
I0208 03:26:39.722746       1 trace.go:76] Trace[1826412938]: "Get /api/v1/namespaces/kube-system/serviceaccounts/default" (started: 2019-02-08 03:26:38.952256294 +0000 UTC m=+12704.661246395) (total time: 770.374424ms):
Trace[1826412938]: [770.327335ms] [770.322685ms] About to write a response
I0208 03:26:39.750771       1 trace.go:76] Trace[1240050532]: "Create /apis/rbac.authorization.k8s.io/v1/clusterrolebindings" (started: 2019-02-08 03:26:37.641610197 +0000 UTC m=+12703.350600271) (total time: 2.109129827s):
Trace[1240050532]: [2.109129827s] [2.098997112s] END
I0208 03:26:39.759151       1 trace.go:76] Trace[787718633]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:39.251154982 +0000 UTC m=+12704.960145062) (total time: 507.687387ms):
Trace[787718633]: [507.687387ms] [507.389175ms] END
I0208 03:26:39.759678       1 trace.go:76] Trace[1012423915]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.authorization.k8s.io/status" (started: 2019-02-08 03:26:39.24993491 +0000 UTC m=+12704.958924992) (total time: 509.63742ms):
Trace[1012423915]: [509.492063ms] [508.463619ms] Object stored in database
E0208 03:26:39.839254       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}
E0208 03:26:39.848646       1 watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted
W0208 03:26:39.853354       1 reflector.go:270] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: watch of *apiextensions.CustomResourceDefinition ended with: The resourceVersion for the provided watch is too old.
E0208 03:26:39.868497       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}
E0208 03:26:40.104263       1 watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted
W0208 03:26:40.105162       1 reflector.go:270] storage/cacher.go:/limitranges: watch of *core.LimitRange ended with: The resourceVersion for the provided watch is too old.
I0208 03:26:40.311269       1 trace.go:76] Trace[436243145]: "GuaranteedUpdate etcd3: *core.Pod" (started: 2019-02-08 03:26:39.77535328 +0000 UTC m=+12705.484343425) (total time: 535.401877ms):
Trace[436243145]: [533.020927ms] [431.669624ms] Transaction committed
I0208 03:26:40.313096       1 trace.go:76] Trace[2040678080]: "Patch /api/v1/namespaces/kube-system/pods/kube-controller-manager-minikube/status" (started: 2019-02-08 03:26:39.725991099 +0000 UTC m=+12705.434981197) (total time: 587.080031ms):
Trace[2040678080]: [136.556589ms] [135.860751ms] About to apply patch
Trace[2040678080]: [585.337704ms] [437.866999ms] Object stored in database
I0208 03:26:40.315762       1 trace.go:76] Trace[1685604219]: "GuaranteedUpdate etcd3: *core.Event" (started: 2019-02-08 03:26:27.303579274 +0000 UTC m=+12693.012569377) (total time: 13.011852053s):
Trace[1685604219]: [12.447999075s] [12.447999075s] initial value restored
Trace[1685604219]: [12.838091254s] [390.092179ms] Transaction prepared
Trace[1685604219]: [13.011707486s] [173.616232ms] Transaction committed
I0208 03:26:40.316163       1 trace.go:76] Trace[1402128361]: "Patch /api/v1/namespaces/kube-system/events/tiller-deploy-6f4dbc6d67-b2rpz.158138a6cfd33236" (started: 2019-02-08 03:26:26.523502843 +0000 UTC m=+12692.232493009) (total time: 13.792441278s):
Trace[1402128361]: [467.240153ms] [467.240153ms] Recorded the audit event
Trace[1402128361]: [13.22807853s] [12.760838377s] About to apply patch
Trace[1402128361]: [13.792280406s] [560.508745ms] Object stored in database
I0208 03:26:40.316819       1 trace.go:76] Trace[240820255]: "GuaranteedUpdate etcd3: *v1.Endpoints" (started: 2019-02-08 03:26:39.725469421 +0000 UTC m=+12705.434459550) (total time: 591.33344ms):
Trace[240820255]: [414.042884ms] [376.323034ms] Transaction prepared
Trace[240820255]: [591.318511ms] [177.275627ms] Transaction committed
I0208 03:26:40.320928       1 trace.go:76] Trace[1362230974]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:39.772442244 +0000 UTC m=+12705.481432328) (total time: 548.459841ms):
Trace[1362230974]: [548.459841ms] [548.074081ms] END
I0208 03:26:40.321418       1 trace.go:76] Trace[1158861642]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.apps/status" (started: 2019-02-08 03:26:39.767162119 +0000 UTC m=+12705.476152208) (total time: 554.240933ms):
Trace[1158861642]: [553.905428ms] [548.656476ms] Object stored in database
I0208 03:26:40.329685       1 trace.go:76] Trace[1785035863]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:39.771197859 +0000 UTC m=+12705.480187974) (total time: 558.460939ms):
Trace[1785035863]: [558.460939ms] [558.403742ms] END
I0208 03:26:40.330740       1 trace.go:76] Trace[507954188]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1./status" (started: 2019-02-08 03:26:39.770655837 +0000 UTC m=+12705.479645903) (total time: 559.096475ms):
Trace[507954188]: [559.054545ms] [558.746004ms] Object stored in database
I0208 03:26:40.362290       1 trace.go:76] Trace[1438690736]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:39.761071489 +0000 UTC m=+12705.470061707) (total time: 601.026211ms):
Trace[1438690736]: [601.026211ms] [600.501798ms] END
I0208 03:26:40.362431       1 trace.go:76] Trace[2142829655]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.authorization.k8s.io/status" (started: 2019-02-08 03:26:39.750943818 +0000 UTC m=+12705.459933910) (total time: 611.474394ms):
Trace[2142829655]: [611.407809ms] [605.337866ms] Object stored in database
I0208 03:26:40.387669       1 trace.go:76] Trace[1109837378]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:39.77594115 +0000 UTC m=+12705.484931246) (total time: 611.468876ms):
Trace[1109837378]: [611.468876ms] [610.22403ms] END
I0208 03:26:40.388260       1 trace.go:76] Trace[281846775]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.batch/status" (started: 2019-02-08 03:26:39.770024095 +0000 UTC m=+12705.479014167) (total time: 618.1787ms):
Trace[281846775]: [617.680684ms] [611.919033ms] Object stored in database
W0208 03:26:40.900460       1 cacher.go:125] Terminating all watchers from cacher *apiextensions.CustomResourceDefinition
W0208 03:26:41.109951       1 cacher.go:125] Terminating all watchers from cacher *core.LimitRange
W0208 03:26:41.494822       1 reflector.go:270] k8s.io/apiextensions-apiserver/pkg/client/informers/internalversion/factory.go:117: watch of *apiextensions.CustomResourceDefinition ended with: too old resource version: 12803 (31679)
I0208 03:26:41.503043       1 trace.go:76] Trace[1633178365]: "Get /api/v1/namespaces/default/endpoints/kubernetes" (started: 2019-02-08 03:26:40.386110884 +0000 UTC m=+12706.095100966) (total time: 1.115377358s):
Trace[1633178365]: [1.11028235s] [1.110106594s] About to write a response
I0208 03:26:41.512745       1 trace.go:76] Trace[1207466851]: "Get /api/v1/namespaces/kube-system/pods/kube-scheduler-minikube" (started: 2019-02-08 03:26:40.433466146 +0000 UTC m=+12706.142456214) (total time: 1.06918354s):
Trace[1207466851]: [1.06595347s] [1.065950068s] About to write a response
I0208 03:26:42.000616       1 trace.go:76] Trace[377774815]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:40.441523058 +0000 UTC m=+12706.150513183) (total time: 1.557255082s):
Trace[377774815]: [1.557255082s] [1.557225248s] END
I0208 03:26:42.003130       1 trace.go:76] Trace[2004817732]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.authentication.k8s.io/status" (started: 2019-02-08 03:26:40.432512088 +0000 UTC m=+12706.141502155) (total time: 1.570588404s):
Trace[2004817732]: [1.568947177s] [1.560104162s] Object stored in database
I0208 03:26:42.014347       1 trace.go:76] Trace[722909859]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:40.440621785 +0000 UTC m=+12706.149611885) (total time: 1.573697492s):
Trace[722909859]: [1.573697492s] [1.573638739s] END
I0208 03:26:42.014995       1 trace.go:76] Trace[1468865881]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.policy/status" (started: 2019-02-08 03:26:40.434354243 +0000 UTC m=+12706.143344309) (total time: 1.580623716s):
Trace[1468865881]: [1.580199538s] [1.574121123s] Object stored in database
I0208 03:26:42.016604       1 trace.go:76] Trace[409652420]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:40.441415451 +0000 UTC m=+12706.150405532) (total time: 1.574654308s):
Trace[409652420]: [1.574654308s] [1.57426571s] END
I0208 03:26:42.017124       1 trace.go:76] Trace[258657736]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.autoscaling/status" (started: 2019-02-08 03:26:40.44048638 +0000 UTC m=+12706.149476466) (total time: 1.576247848s):
Trace[258657736]: [1.576196216s] [1.575444083s] Object stored in database
I0208 03:26:42.051603       1 trace.go:76] Trace[1073573133]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:40.427659245 +0000 UTC m=+12706.136649367) (total time: 1.623917423s):
Trace[1073573133]: [1.623917423s] [1.623844396s] END
I0208 03:26:42.052304       1 trace.go:76] Trace[262265716]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.storage.k8s.io/status" (started: 2019-02-08 03:26:40.42676345 +0000 UTC m=+12706.135753523) (total time: 1.62512423s):
Trace[262265716]: [1.625048612s] [1.624467122s] Object stored in database
I0208 03:26:42.063060       1 trace.go:76] Trace[1905920047]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:40.433823997 +0000 UTC m=+12706.142814094) (total time: 1.583681065s):
Trace[1905920047]: [1.583681065s] [1.583088244s] END
I0208 03:26:42.063486       1 trace.go:76] Trace[1997348401]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.scheduling.k8s.io/status" (started: 2019-02-08 03:26:40.431043178 +0000 UTC m=+12706.140033244) (total time: 1.632419181s):
Trace[1997348401]: [1.632183716s] [1.629689416s] Object stored in database
I0208 03:26:42.085156       1 trace.go:76] Trace[480274450]: "Create /apis/storage.k8s.io/v1/storageclasses" (started: 2019-02-08 03:26:39.906568069 +0000 UTC m=+12705.615558145) (total time: 2.178562074s):
Trace[480274450]: [2.178562074s] [2.067165964s] END
I0208 03:26:45.183007       1 trace.go:76] Trace[901306837]: "GuaranteedUpdate etcd3: *core.Pod" (started: 2019-02-08 03:26:41.579417927 +0000 UTC m=+12707.288408048) (total time: 3.602841682s):
Trace[901306837]: [3.598606979s] [3.518497483s] Transaction committed
I0208 03:26:45.196788       1 trace.go:76] Trace[1214165422]: "Patch /api/v1/namespaces/kube-system/pods/kube-scheduler-minikube/status" (started: 2019-02-08 03:26:41.571033882 +0000 UTC m=+12707.280023945) (total time: 3.625543282s):
Trace[1214165422]: [83.105759ms] [73.660211ms] About to check admission control
Trace[1214165422]: [3.624482714s] [3.541376955s] Object stored in database
I0208 03:26:45.197262       1 trace.go:76] Trace[984161982]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:42.229401644 +0000 UTC m=+12707.938391858) (total time: 2.858405729s):
Trace[984161982]: [2.858405729s] [2.852283448s] END
I0208 03:26:45.266812       1 trace.go:76] Trace[824580336]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:42.23722932 +0000 UTC m=+12707.946219398) (total time: 3.029554416s):
Trace[824580336]: [3.029554416s] [3.029519774s] END
I0208 03:26:45.276369       1 trace.go:76] Trace[1347390405]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.extensions/status" (started: 2019-02-08 03:26:42.130142799 +0000 UTC m=+12707.839132859) (total time: 3.145166296s):
Trace[1347390405]: [107.008153ms] [107.008153ms] About to convert to expected version
Trace[1347390405]: [3.136762952s] [3.029713946s] Object stored in database
I0208 03:26:45.285900       1 trace.go:76] Trace[1648766121]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.storage.k8s.io/status" (started: 2019-02-08 03:26:42.08304024 +0000 UTC m=+12707.792030328) (total time: 3.202587989s):
Trace[1648766121]: [3.11454036s] [3.113026381s] Object stored in database
Trace[1648766121]: [3.202587989s] [87.871349ms] END
W0208 03:26:45.300667       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.LimitRange ended with: too old resource version: 12803 (31681)
I0208 03:26:45.371431       1 trace.go:76] Trace[1648713318]: "Get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:26:43.479949705 +0000 UTC m=+12709.188939772) (total time: 1.891252622s):
Trace[1648713318]: [1.890189386s] [1.889754479s] About to write a response
I0208 03:26:45.374999       1 trace.go:76] Trace[574596545]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:42.13263056 +0000 UTC m=+12707.841620660) (total time: 3.242345107s):
Trace[574596545]: [3.242345107s] [3.241588198s] END
I0208 03:26:45.377118       1 trace.go:76] Trace[1449547322]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.admissionregistration.k8s.io/status" (started: 2019-02-08 03:26:42.132327861 +0000 UTC m=+12707.841317927) (total time: 3.244767933s):
Trace[1449547322]: [3.244497898s] [3.24445496s] Object stored in database
I0208 03:26:45.380927       1 trace.go:76] Trace[296324637]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:42.131603178 +0000 UTC m=+12707.840593252) (total time: 3.249298668s):
Trace[296324637]: [3.249298668s] [3.247624724s] END
I0208 03:26:45.382106       1 trace.go:76] Trace[1126929435]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.apps/status" (started: 2019-02-08 03:26:42.130983911 +0000 UTC m=+12707.839973982) (total time: 3.2509515s):
Trace[1126929435]: [3.250421804s] [3.249842773s] Object stored in database
I0208 03:26:45.383182       1 trace.go:76] Trace[1280530598]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:42.237466143 +0000 UTC m=+12707.946456250) (total time: 3.145696328s):
Trace[1280530598]: [3.145696328s] [3.145668496s] END
I0208 03:26:45.383469       1 trace.go:76] Trace[17786595]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.authentication.k8s.io/status" (started: 2019-02-08 03:26:42.129229553 +0000 UTC m=+12707.838219621) (total time: 3.254224161s):
Trace[17786595]: [3.254080548s] [3.201605333s] Object stored in database
I0208 03:26:45.593944       1 trace.go:76] Trace[1228764129]: "GuaranteedUpdate etcd3: *core.Event" (started: 2019-02-08 03:26:40.369590641 +0000 UTC m=+12706.078581004) (total time: 5.223244554s):
Trace[1228764129]: [1.683905381s] [1.683905381s] initial value restored
Trace[1228764129]: [5.222623804s] [3.532063601s] Transaction committed
I0208 03:26:45.784842       1 trace.go:76] Trace[118585738]: "Patch /api/v1/namespaces/kube-system/events/kube-apiserver-minikube.158138a976c0bae6" (started: 2019-02-08 03:26:40.367207107 +0000 UTC m=+12706.076197179) (total time: 5.417137131s):
Trace[118585738]: [1.686293043s] [1.685900836s] About to apply patch
Trace[118585738]: [5.226950675s] [3.537802168s] Object stored in database
Trace[118585738]: [5.417137131s] [189.720833ms] END
I0208 03:26:46.113608       1 trace.go:76] Trace[1766067354]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:45.391826292 +0000 UTC m=+12711.100816399) (total time: 721.565009ms):
Trace[1766067354]: [721.565009ms] [719.780209ms] END
I0208 03:26:46.114074       1 trace.go:76] Trace[1998007416]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v2beta1.autoscaling/status" (started: 2019-02-08 03:26:45.3773242 +0000 UTC m=+12711.086314290) (total time: 736.732111ms):
Trace[1998007416]: [736.31739ms] [723.261194ms] Object stored in database
I0208 03:26:46.114540       1 trace.go:76] Trace[60293929]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:45.400146235 +0000 UTC m=+12711.109136351) (total time: 714.378465ms):
Trace[60293929]: [714.378465ms] [714.317562ms] END
I0208 03:26:46.115174       1 trace.go:76] Trace[465114153]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v2beta2.autoscaling/status" (started: 2019-02-08 03:26:45.398729561 +0000 UTC m=+12711.107719624) (total time: 716.324173ms):
Trace[465114153]: [715.835793ms] [714.737163ms] Object stored in database
I0208 03:26:46.415976       1 trace.go:76] Trace[1542090766]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:45.780069583 +0000 UTC m=+12711.489059671) (total time: 635.870295ms):
Trace[1542090766]: [635.870295ms] [635.803562ms] END
I0208 03:26:46.416318       1 trace.go:76] Trace[2130703486]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.networking.k8s.io/status" (started: 2019-02-08 03:26:45.725140184 +0000 UTC m=+12711.434130248) (total time: 691.13707ms):
Trace[2130703486]: [690.986412ms] [636.225675ms] Object stored in database
I0208 03:26:46.424139       1 trace.go:76] Trace[608345469]: "Get /api/v1/namespaces/kube-system/secrets/storage-provisioner-token-rlnmf" (started: 2019-02-08 03:26:45.787918081 +0000 UTC m=+12711.496908153) (total time: 636.057064ms):
Trace[608345469]: [635.115939ms] [635.109374ms] About to write a response
I0208 03:26:46.471783       1 trace.go:76] Trace[2078584805]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-02-08 03:26:45.733121487 +0000 UTC m=+12711.442111666) (total time: 738.450837ms):
Trace[2078584805]: [738.450837ms] [738.387842ms] END
I0208 03:26:46.472681       1 trace.go:76] Trace[1196438747]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.coordination.k8s.io/status" (started: 2019-02-08 03:26:45.731161425 +0000 UTC m=+12711.440151570) (total time: 741.294668ms):
Trace[1196438747]: [740.751123ms] [740.364221ms] Object stored in database
I0208 03:26:47.701110       1 trace.go:76] Trace[1987640049]: "GuaranteedUpdate etcd3: *core.Event" (started: 2019-02-08 03:26:47.128015324 +0000 UTC m=+12712.837005402) (total time: 570.419162ms):
Trace[1987640049]: [363.044587ms] [363.044587ms] initial value restored
Trace[1987640049]: [570.359854ms] [206.597811ms] Transaction committed
I0208 03:26:47.702679       1 trace.go:76] Trace[2006474077]: "Patch /api/v1/namespaces/kube-system/events/kube-scheduler-minikube.1580f6df3f5aef84" (started: 2019-02-08 03:26:47.12772616 +0000 UTC m=+12712.836716231) (total time: 574.926075ms):
Trace[2006474077]: [363.337316ms] [363.312179ms] About to apply patch
Trace[2006474077]: [573.948293ms] [210.27291ms] Object stored in database
I0208 03:26:48.779031       1 trace.go:76] Trace[2008937990]: "Get /api/v1/namespaces/kube-system/serviceaccounts/storage-provisioner" (started: 2019-02-08 03:26:48.231802972 +0000 UTC m=+12713.940793056) (total time: 546.856645ms):
Trace[2008937990]: [545.667626ms] [545.661846ms] About to write a response
I0208 03:26:50.298629       1 trace.go:76] Trace[1357358342]: "GuaranteedUpdate etcd3: *core.Node" (started: 2019-02-08 03:26:49.53745562 +0000 UTC m=+12715.246445828) (total time: 759.188432ms):
Trace[1357358342]: [747.893473ms] [727.335986ms] Transaction committed
I0208 03:26:50.301469       1 trace.go:76] Trace[1919748199]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:26:49.775396035 +0000 UTC m=+12715.484386099) (total time: 525.849904ms):
Trace[1919748199]: [525.560625ms] [525.357602ms] About to write a response
I0208 03:26:50.312259       1 trace.go:76] Trace[297661504]: "List /api/v1/namespaces/kube-system/services" (started: 2019-02-08 03:26:49.652841131 +0000 UTC m=+12715.361831245) (total time: 658.929769ms):
Trace[297661504]: [653.962288ms] [653.126104ms] Listing from storage done
I0208 03:26:50.387857       1 trace.go:76] Trace[1721666097]: "Patch /api/v1/nodes/minikube/status" (started: 2019-02-08 03:26:49.535015281 +0000 UTC m=+12715.244005355) (total time: 852.621897ms):
Trace[1721666097]: [764.458783ms] [748.117756ms] Object stored in database
Trace[1721666097]: [852.028498ms] [87.559621ms] Writing response
I0208 03:27:11.361319       1 trace.go:76] Trace[509809736]: "List /apis/events.k8s.io/v1beta1/events" (started: 2019-02-08 03:27:10.776203866 +0000 UTC m=+12736.485193937) (total time: 583.099358ms):
Trace[509809736]: [580.747702ms] [580.721342ms] Listing from storage done
I0208 03:27:11.630810       1 trace.go:76] Trace[1280968516]: "List /api/v1/nodes" (started: 2019-02-08 03:27:11.022099207 +0000 UTC m=+12736.731089270) (total time: 608.682283ms):
Trace[1280968516]: [350.40466ms] [350.400191ms] Listing from storage done
Trace[1280968516]: [608.678857ms] [258.123635ms] Writing http response done (1 items)
I0208 03:27:11.947658       1 trace.go:76] Trace[450103235]: "Update /api/v1/namespaces/local-chain/endpoints/accelerator-master" (started: 2019-02-08 03:27:11.445318496 +0000 UTC m=+12737.154308556) (total time: 502.154848ms):
Trace[450103235]: [188.657901ms] [188.657901ms] About to convert to expected version
Trace[450103235]: [502.111674ms] [313.425692ms] Object stored in database
I0208 03:27:11.948383       1 trace.go:76] Trace[1203401686]: "Get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:27:11.367441237 +0000 UTC m=+12737.076431298) (total time: 580.922182ms):
Trace[1203401686]: [580.891704ms] [580.86438ms] About to write a response
I0208 03:27:11.948828       1 trace.go:76] Trace[1415994295]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:27:11.364271765 +0000 UTC m=+12737.073261873) (total time: 584.540041ms):
Trace[1415994295]: [584.540041ms] [584.508307ms] END
I0208 03:27:11.949016       1 trace.go:76] Trace[1003127388]: "Update /api/v1/namespaces/local-chain/endpoints/accelerator" (started: 2019-02-08 03:27:11.350179872 +0000 UTC m=+12737.059169934) (total time: 598.822203ms):
Trace[1003127388]: [598.777053ms] [584.926868ms] Object stored in database
I0208 03:27:23.371616       1 trace.go:76] Trace[1555656188]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:27:22.796566855 +0000 UTC m=+12748.505556943) (total time: 572.214986ms):
Trace[1555656188]: [571.891513ms] [549.135907ms] Transaction committed
I0208 03:27:23.374548       1 trace.go:76] Trace[139730317]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:27:22.6644983 +0000 UTC m=+12748.373488392) (total time: 710.017611ms):
Trace[139730317]: [106.138233ms] [106.138233ms] About to convert to expected version
Trace[139730317]: [707.655761ms] [593.702474ms] Object stored in database
I0208 03:27:25.818638       1 trace.go:76] Trace[674477648]: "Get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:27:24.97754497 +0000 UTC m=+12750.686535056) (total time: 839.700957ms):
Trace[674477648]: [836.959835ms] [836.947512ms] About to write a response
I0208 03:27:26.052285       1 trace.go:76] Trace[1159251151]: "GuaranteedUpdate etcd3: *core.Event" (started: 2019-02-08 03:27:24.919785294 +0000 UTC m=+12750.628775389) (total time: 1.132369446s):
Trace[1159251151]: [1.031976361s] [1.031976361s] initial value restored
I0208 03:27:26.052484       1 trace.go:76] Trace[943090685]: "Patch /api/v1/namespaces/kube-system/events/tiller-deploy-6f4dbc6d67-b2rpz.158138a8e79e4deb" (started: 2019-02-08 03:27:24.914150104 +0000 UTC m=+12750.623140176) (total time: 1.138314767s):
Trace[943090685]: [1.037948767s] [1.036804948s] About to apply patch
Trace[943090685]: [1.138171846s] [94.629669ms] Object stored in database
I0208 03:27:27.470087       1 trace.go:76] Trace[479324897]: "GuaranteedUpdate etcd3: *v1.Endpoints" (started: 2019-02-08 03:27:26.632282664 +0000 UTC m=+12752.341272745) (total time: 835.088644ms):
Trace[479324897]: [479.96369ms] [393.212041ms] Transaction prepared
Trace[479324897]: [832.711102ms] [352.747412ms] Transaction committed
I0208 03:27:28.199700       1 trace.go:76] Trace[1302164168]: "Get /api/v1/namespaces/default/endpoints/kubernetes" (started: 2019-02-08 03:27:27.513019441 +0000 UTC m=+12753.222009549) (total time: 683.941962ms):
Trace[1302164168]: [683.606043ms] [683.460308ms] About to write a response
I0208 03:27:28.841531       1 trace.go:76] Trace[35027963]: "List /apis/batch/v1/jobs" (started: 2019-02-08 03:27:27.858737 +0000 UTC m=+12753.567727069) (total time: 981.166826ms):
Trace[35027963]: [972.131581ms] [972.121834ms] Listing from storage done
I0208 03:27:29.065613       1 trace.go:76] Trace[1106686680]: "Get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:27:28.300890888 +0000 UTC m=+12754.009880965) (total time: 764.43079ms):
Trace[1106686680]: [764.094366ms] [745.268334ms] About to write a response
I0208 03:27:29.110008       1 trace.go:76] Trace[1990129949]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-02-08 03:27:28.331066102 +0000 UTC m=+12754.040056173) (total time: 778.912035ms):
Trace[1990129949]: [778.844032ms] [778.80556ms] About to write a response
I0208 03:27:29.952672       1 trace.go:76] Trace[1590018136]: "Create /apis/rbac.authorization.k8s.io/v1/clusterrolebindings" (started: 2019-02-08 03:27:28.189384395 +0000 UTC m=+12753.898374463) (total time: 1.692710396s):
Trace[1590018136]: [150.865945ms] [150.865945ms] About to convert to expected version
Trace[1590018136]: [1.692710396s] [1.528014032s] END
I0208 03:27:30.418130       1 trace.go:76] Trace[1370249491]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-02-08 03:27:29.412215721 +0000 UTC m=+12755.121205822) (total time: 1.003376766s):
Trace[1370249491]: [1.003341615s] [884.490652ms] Transaction committed
I0208 03:27:30.546844       1 trace.go:76] Trace[495934776]: "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started: 2019-02-08 03:27:29.311140726 +0000 UTC m=+12755.020130794) (total time: 1.235374142s):
Trace[495934776]: [1.107419707s] [1.065596048s] Object stored in database
Trace[495934776]: [1.235374142s] [126.491724ms] END

@kamilgregorczyk
Copy link

The same thing, I'm getting these errors over and over after some time which is very frustrating. I don't do anything special except running minikube helm & istio. It's also not a resource problem as I assigned 8 cores and 20gb of ram.

@andrewgdavis
Copy link

Very similar setup as @kamilgregorczyk; with helm and istio running on osx (virtualbox). i started minikube with 6 cpu and 8192 of ram; but it looks like the apiserver gets overloaded.
it takes a little over a minute and a half for it to start listening...

first log message:
I0305 18:28:28.399073       1 server.go:557] external host was not specified, using 192.168.99.100
...
...
I0305 18:30:06.875123       1 establishing_controller.go:73] Starting EstablishingController

and afterward its followed up with a bunch of TLS handshake errors that look like this:

I0305 18:30:06.907062       1 log.go:172] http: TLS handshake error from 172.17.0.12:54836: EOF

then more errors that say 8443 connect: connection refused... and the last couple messages before the container restarts is:

I0305 18:44:24.456725       1 trace.go:76] Trace[1799555601]: "GuaranteedUpdate etcd3: *core.RangeAllocation" (started: 2019-03-05 18:43:59.616177608 +0000 UTC m=+123.088015866) (total time: 24.840516435s):
Trace[1799555601]: [24.840302421s] [24.840302421s] initial value restored
E0305 18:44:24.498355       1 controller.go:155] Unable to perform initial Kubernetes service initialization: Post https://[::1]:8443/api/v1/namespaces: dial tcp [::1]:8443: connect: connection refused
E0305 18:44:24.585057       1 controller.go:204] unable to sync kubernetes service: Post https://[::1]:8443/api/v1/namespaces: dial tcp [::1]:8443: connect: connection refused
E0305 18:44:24.585120       1 controller.go:191] unable to create required kubernetes system namespace kube-system: Post https://[::1]:8443/api/v1/namespaces: dial tcp [::1]:8443: connect: connection refused
E0305 18:44:24.612777       1 controller.go:191] unable to create required kubernetes system namespace kube-public: Post https://[::1]:8443/api/v1/namespaces: dial tcp [::1]:8443: connect: connection refused
E0305 18:44:25.073390       1 repair.go:74] unable to refresh the port block: Get https://[::1]:8443/api/v1/services: dial tcp [::1]:8443: connect: connection refused
E0305 18:44:25.576473       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}

@Timoses
Copy link

Timoses commented Jun 13, 2019

Same here running minikube on Manjaro with kvm2 driver.

Only running Istio on the cluster.

Kubernetes v1.13.0 and Istio 1.1.8

@Timoses
Copy link

Timoses commented Jun 13, 2019

Think I fixed this one by applying more memory to the minikube vm as documented here: https://istio.io/docs/setup/kubernetes/prepare/platform-setup/minikube/

@kamilgregorczyk
Copy link

I applied 20gbs..

@deatherving
Copy link

@kamilgregorczyk Did you set the configuration in Virtual Box?

@kamilgregorczyk
Copy link

Yes

@lienharc
Copy link

lienharc commented Nov 5, 2019

Same here with 12 GB memory

@adammil2000
Copy link

I am having the same problem using Kubernetes on minikube. The CPU is cycling up to 100% as the apiserver is cycling between status Running, Error, and Stopped. I have allocated 16GB and 4 CPUs to Docker and changing this up and down doesn't affect the problem. I can't even run the kubectl get pods command without getting back one of these two errors, seems to depend on the state of apiserver at the time:

Error from server: rpc error: code = Unavailable desc = transport is closing
The connection to the server 192.168.64.2:8443 was refused - did you specify the right host or port?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests