fails with -extra-config=apiserver.authorization-mode=RBAC and audit logging: timed out waiting for kube-proxy

[ixtendio]

BUG REPORT

Environment:

Minikube version (use minikube version): v0.28.0

• OS (e.g. from /etc/os-release): MacOS High Sierra 10.13.5
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): VirtualBox 5.2.12 r122591 (Qt5.6.3)
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): Boot2DockerURL": "file:///Users/123456/.minikube/cache/iso/minikube-v0.28.0.iso
• Install tools: curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
  minikube.log

What happened:

When I try to start minikube using one of these commands:

minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.LogOptions.Path=/var/log/audit.log --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml

or

minikube start --extra-config=apiserver.authorization-mode=RBAC --extra-config=apiserver.audit-log-path=/var/log/audit.log --extra-config=apiserver.audit-policy-file=/etc/kubernetes/addons/audit-policy.yaml

The content of audit-policy.yaml being:

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata

it's end up by hanging the minikube in the line: Starting cluster components... and in the end I get this error:
E0625 21:34:15.410946 54396 start.go:299] Error restarting cluster: restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition

What you expected to happen: to start minikube with the audit logging

How to reproduce it (as minimally and precisely as possible): stop the minikube and start it using one of the above commands.

Output of minikube logs (if applicable): minikube.log

[Rouche]

I have the exact same problem.

Minikube version: v0.28.0
VM Driver: "DriverName": "virtualbox",
ISO Version: "Boot2DockerURL": "file://C:/Users/resolutech/.minikube/cache/iso/minikube-v0.28.0.iso"
Kubernetes: 1.10.0
VirtualBox: 5.2.12
OS: Windows 10 PRO Build 17134.rs4

With minikube start Everything is working.

Using "Minimal policy file" here: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata

Comparing working log with failing log (same command), i have those lines in the failing one:

Jun 25 19:46:38 minikube kubelet[2681]: I0625 19:46:38.841304    2681 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
Jun 25 19:46:38 minikube kubelet[2681]: I0625 19:46:38.843612    2681 kubelet_node_status.go:82] Attempting to register node minikube
Jun 25 19:46:38 minikube kubelet[2681]: E0625 19:46:38.843987    2681 kubelet_node_status.go:106] Unable to register node "minikube" with API server: Post https://192.168.99.100:8443/api/v1/nodes: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 25 19:46:39 minikube kubelet[2681]: E0625 19:46:39.048114    2681 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.99.100:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 25 19:46:39 minikube kubelet[2681]: E0625 19:46:39.049432    2681 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.99.100:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 25 19:46:39 minikube kubelet[2681]: E0625 19:46:39.054768    2681 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.99.100:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Jun 25 19:46:39 minikube kubelet[2681]: E0625 19:46:39.234038    2681 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "minikube" not found

Note: I try to follow a course here: https://www.udemy.com/kubernetes-from-a-devops-kubernetes-guru/
But examples are identical to Kubernetes documentation. I also double checked apiserver source code to make sure Flags were ok: https://github.com/kubernetes/minikube/tree/v0.28.0/vendor/k8s.io

[kairen]

Hi @tavyy and @Rouche, I added #2951 to support feature gate for Kubeadm. This feature can help you to enable auditing log in Minikube:

$ minikube start --kubeadm-feature-gates=Auditing=true

[Rouche]

Thanks.

Hopefully it gets merged for next version. never tryed to build minikube "yet" to test PR.

[cxhercules]

@kairen how did you manage to place the audit file in /etc/kubernetes/audit/audit.yaml? Also I still get the timeout, and I am using version v0.28.2.

I see this in logs:

Aug 10 21:17:04 minikube kubelet[2737]: E0810 21:17:04.897435    2737 kubelet_node_status.go:106] Unable to register node "minikube"
with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Aug 10 21:17:05 minikube kubelet[2737]: E0810 21:17:05.394666    2737 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460:
 Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0:
 dial tcp 127.0.0.1:8443: getsockopt: connection refused
Aug 10 21:17:05 minikube kubelet[2737]: E0810 21:17:05.395483    2737 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserve
r.go:47: Failed to list *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVers
ion=0: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Aug 10 21:17:05 minikube kubelet[2737]: E0810 21:17:05.397465    2737 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451:
 Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: getsock
opt: connection refused
Aug 10 21:17:06 minikube kubelet[2737]: E0810 21:17:06.185236    2737 eviction_manager.go:246] eviction manager: failed to get get su
mmary stats: failed to get node info: node "minikube" not found
Aug 10 21:17:06 minikube kubelet[2737]: E0810 21:17:06.396190    2737 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460:
 Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0:
 dial tcp 127.0.0.1:8443: getsockopt: connection refused
Aug 10 21:17:06 minikube kubelet[2737]: E0810 21:17:06.397382    2737 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserve

[tstromberg]

Probably a dupe of #2852 - please re-open if not.