Add minikube annotation to kubeconfig context and cluster extensions

[briandealwis]

Running minikube profile list -ojson is surprisingly slow, taking up to 2.5s on my 2016 MBP. In part this may be due to my having a second profile from gcloud code dev.

This is relevant as Skaffold currently invokes minikube profile list to determine if it's building against Minikube. (We actually use a cheaper heuristic by first checking if the context name is minikube, but that will be no longer possible now that people are experimenting with running Minikube with containerd as its container runtime GoogleContainerTools/skaffold#5143).

Our use of minikube profile list causes a noticeable delay when deploying to a remote cluster.

Steps to reproduce the issue:

1. minikube profile list -ojson

Full output of failed command:

$  time minikube profile list -ojson --alsologtostderr
I1215 15:08:27.294810   91926 out.go:185] Setting OutFile to fd 1 ...
I1215 15:08:27.295371   91926 out.go:237] isatty.IsTerminal(1) = true
I1215 15:08:27.295381   91926 out.go:198] Setting ErrFile to fd 2...
I1215 15:08:27.295387   91926 out.go:237] isatty.IsTerminal(2) = true
I1215 15:08:27.295470   91926 root.go:279] Updating PATH: /Users/bdealwis/.minikube/bin
I1215 15:08:27.296589   91926 cli_runner.go:110] Run: docker ps -a --filter label=name.minikube.sigs.k8s.io --format {{.Names}}
I1215 15:08:27.456061   91926 host.go:66] Checking if "gcloud-local-dev" exists ...
I1215 15:08:27.457050   91926 cli_runner.go:110] Run: docker container inspect gcloud-local-dev --format={{.State.Status}}
W1215 15:08:27.617335   91926 cli_runner.go:148] docker container inspect gcloud-local-dev --format={{.State.Status}} returned with exit code 1
W1215 15:08:27.618631   91926 profile_list.go:104] error getting host state: unknown state "gcloud-local-dev": docker container inspect gcloud-local-dev --format={{.State.Status}}: exit status 1
stdout:


stderr:
Error: No such container: gcloud-local-dev
I1215 15:08:27.618664   91926 host.go:66] Checking if "minikube" exists ...
I1215 15:08:27.619629   91926 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Status}}
I1215 15:08:27.780332   91926 cli_runner.go:110] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" minikube
I1215 15:08:27.940738   91926 api_server.go:146] Checking apiserver status ...
I1215 15:08:27.941009   91926 ssh_runner.go:148] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I1215 15:08:27.941175   91926 cli_runner.go:110] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I1215 15:08:28.096726   91926 sshutil.go:45] new ssh client: &{IP:127.0.0.1 Port:49156 SSHKeyPath:/Users/bdealwis/.minikube/machines/minikube/id_rsa Username:docker}
I1215 15:08:28.242264   91926 ssh_runner.go:148] Run: sudo egrep ^[0-9]+:freezer: /proc/125734/cgroup
I1215 15:08:28.262518   91926 api_server.go:162] apiserver freezer: "7:freezer:/docker/a17a973c1edf1b02d275c326bb3850c15cbf38f1bca50f4268f78aa12504e583/kubepods/burstable/podf7c3d51df5e2ce4e433b64661ac4503c/a8bd56289e07a30a76905b89c6b6c5c6fdc7a7f0b97f9910cef2f599e7e324e2"
I1215 15:08:28.262695   91926 ssh_runner.go:148] Run: sudo cat /sys/fs/cgroup/freezer/docker/a17a973c1edf1b02d275c326bb3850c15cbf38f1bca50f4268f78aa12504e583/kubepods/burstable/podf7c3d51df5e2ce4e433b64661ac4503c/a8bd56289e07a30a76905b89c6b6c5c6fdc7a7f0b97f9910cef2f599e7e324e2/freezer.state
I1215 15:08:28.282210   91926 api_server.go:184] freezer state: "THAWED"
I1215 15:08:28.282253   91926 api_server.go:221] Checking apiserver healthz at https://127.0.0.1:49153/healthz ...
I1215 15:08:28.295026   91926 api_server.go:241] https://127.0.0.1:49153/healthz returned 200:
ok
I1215 15:08:28.313899   91926 out.go:110] {"invalid":[],"valid":[{"Name":"gcloud-local-dev","Status":"Unknown","Config":{"Name":"gcloud-local-dev","KeepContext":true,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.12-snapshot3@sha256:1d687ba53e19dbe5fafe4cc18aa07f269ecc4b7b622f2251b5bf569ddb474e9b","Memory":2949,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","KubernetesConfig":{"KubernetesVersion":"v1.19.0","ClusterName":"gcloud-local-dev","Namespace":"","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"172.17.0.3","Port":8443,"KubernetesVersion":"v1.19.0","ControlPlane":true,"Worker":true}],"Addons":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":null}},{"Name":"minikube","Status":"Running","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f","Memory":2944,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","KubernetesConfig":{"KubernetesVersion":"v1.19.2","ClusterName":"minikube","Namespace":"","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"192.168.49.2","Port":8443,"KubernetesVersion":"v1.19.2","ControlPlane":true,"Worker":true}],"Addons":{"ambassador":false,"csi-hostpath-driver":false,"dashboard":false,"default-storageclass":true,"efk":false,"freshpod":false,"gcp-auth":false,"gvisor":false,"helm-tiller":false,"ingress":false,"ingress-dns":false,"istio":false,"istio-provisioner":false,"kubevirt":false,"logviewer":false,"metallb":false,"metrics-server":false,"nvidia-driver-installer":false,"nvidia-gpu-device-plugin":false,"olm":false,"pod-security-policy":false,"registry":false,"registry-aliases":false,"registry-creds":false,"storage-provisioner":true,"storage-provisioner-gluster":false,"volumesnapshots":false},"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":[]}}]}
{"invalid":[],"valid":[{"Name":"gcloud-local-dev","Status":"Unknown","Config":{"Name":"gcloud-local-dev","KeepContext":true,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.12-snapshot3@sha256:1d687ba53e19dbe5fafe4cc18aa07f269ecc4b7b622f2251b5bf569ddb474e9b","Memory":2949,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","KubernetesConfig":{"KubernetesVersion":"v1.19.0","ClusterName":"gcloud-local-dev","Namespace":"","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"172.17.0.3","Port":8443,"KubernetesVersion":"v1.19.0","ControlPlane":true,"Worker":true}],"Addons":null,"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":null}},{"Name":"minikube","Status":"Running","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f","Memory":2944,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":null,"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","KubernetesConfig":{"KubernetesVersion":"v1.19.2","ClusterName":"minikube","Namespace":"","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"","LoadBalancerStartIP":"","LoadBalancerEndIP":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"192.168.49.2","Port":8443,"KubernetesVersion":"v1.19.2","ControlPlane":true,"Worker":true}],"Addons":{"ambassador":false,"csi-hostpath-driver":false,"dashboard":false,"default-storageclass":true,"efk":false,"freshpod":false,"gcp-auth":false,"gvisor":false,"helm-tiller":false,"ingress":false,"ingress-dns":false,"istio":false,"istio-provisioner":false,"kubevirt":false,"logviewer":false,"metallb":false,"metrics-server":false,"nvidia-driver-installer":false,"nvidia-gpu-device-plugin":false,"olm":false,"pod-security-policy":false,"registry":false,"registry-aliases":false,"registry-creds":false,"storage-provisioner":true,"storage-provisioner-gluster":false,"volumesnapshots":false},"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000,"ScheduledStop":null,"ExposedPorts":[]}}]}
minikube profile list -ojson --alsologtostderr  0.62s user 0.39s system 93% cpu 1.082 total

[spowelljr]

/priority important-soon

[medyagh]

@briandealwis the reason seems to be high CPU usage
profile list checks for the status of the container or VM so this could take time if CPU usage is high or docker is not pruned.
as on option, we could provide a shorter profile list without "Status" sometihng like
"minikube profile list --short"

would that be a good solution or you would need the status ?

[medyagh]

per conversation with @briandealwis on slack we came up this solution to add informations in "extension" block of the kubeconfig to make minikube clusters identifiable by kubeconfig