--extra-config doesn't work on preexisting clusters

[franck102]

The problem started appearing yesterday, stopping / restarting minikube or deleting the apiserver pod doesn't help.
The VM hogs the CPU, and connections to the apiserver are failing (esp. a CreateWatch call from a Java client)

1. Start minikube with
   /usr/local/bin/minikube [--extra-config=apiserver.v=1] --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind-address=0.0.0.0 start
   (same result with or without the apiserver.v=1 extra config)

2. kubectl -n kube-system logs

3. Entries are logged at the Trace level: 5mn after starting the cluster the logs have 2300 lines, and VirtualBox uses 284% CPU

Full output of minikube start command used, if not already included:
😄 minikube v1.11.0 on Darwin 10.14.1
▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨ Using the virtualbox driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing virtualbox VM for "minikube" ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
▪ apiserver.v=1
▪ kubelet.authentication-token-webhook=true
▪ kubelet.authorization-mode=Webhook
▪ scheduler.bind-address=0.0.0.0
▪ controller-manager.bind-address=0.0.0.0
🔎 Verifying Kubernetes components...
🌟 Enabled addons: dashboard, default-storageclass, ingress, metrics-server, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"

❗ /usr/local/bin/kubectl is version 1.15.2, which may be incompatible with Kubernetes 1.18.3.
💡 You can also use 'minikube kubectl -- get pods' to invoke a matching version

** minikube config
fma$ minikube config view

• vm-driver: virtualbox
• WantReportError: true
• coredns: false
• disk-size: 30G
• ingress: true
• kube-dns: true
• memory: 16384
• metrics-server: true
• cpus: 6
• dashboard: true
• heapster: true

[medyagh]

@franck102 thank you for reporting this, it does appear to be a bug. while I am not sure the root cause I would be happy to review a PR that fixes this

btw are you sure the format of the extra option is correct I am not familiar with [--extra-config=apiserver.v=1]

and is there a reason you put that on in a bracket [ ] ?

and also could you provide me a way to verify this myself, what is the expected result if you set this paramter?

[medyagh]

/triage needs-information
/triage support

[franck102]

@franck102 thank you for reporting this, it does appear to be a bug. while I am not sure the root cause I would be happy to review a PR that fixes this

btw are you sure the format of the extra option is correct I am not familiar with [--extra-config=apiserver.v=1]

and is there a reason you put that on in a bracket [ ] ?

and also could you provide me a way to verify this myself, what is the expected result if you set this paramter?

I added the brackets to indicate the result was the same with or without the switch:

/usr/local/bin/minikube --extra-config=apiserver.v=1 ...   and
/usr/local/bin/minikube ...

... both give the same result.

To verify, I would expect that with the -v=1 flag supplied to the apiserver, trace entries like the one below would not appear in the pod's logs anymore:

> kc -n kube-system logs kube-apiserver-minikube
I0708 14:14:11.064157       1 trace.go:116] Trace[829415145]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/minikube,user-agent:kubelet/v1.18.3 (linux/amd64) kubernetes/2e7996e,client:192.168.99.100 (started: 2020-07-08 14:14:10.422290961 +0000 UTC m=+2407.352953889) (total time: 641.849249ms):
Trace[829415145]: [641.803537ms] [641.737383ms] Object stored in database

[lachee10]

I had a similar issue yesterday. Maybe the root cause is the same.

Can you check your ~/.minikube/profiles/minikube/config.json file? There should be an section named "ExtraOptions" which contains thec. If it doesn't contain your apiserver.v=1 option then it looks like the same root cause.

In my case I added a few "--extra-config" flags later on after I started minikube for the first time with a "--extra-config" flag. It never used those new flags until I changed the config.json file. It seems the 'minikube' command does never pickup those additional flags you add or remove after having provided some flag. You can even use 'minikube start' and it will use those old flags from the config.json file.

[prasadkatti]

Is this a dup of #8242?

[franck102]

It may be, but I don’t know.
Either no extra-flags are ever passed because the entire mechanism is broken.
Or some work and some don’t... can’t tell without looking into the source code...

[priyawadhwa]

Hey @franck102 I tried to repro this with minikube v1.12.0 by running:

$ minikube delete
$ minikube start --extra-config apiserver.v=1

I confirmed that this worked as expected by running:

$ minikube ssh
$ ps aux | grep v=1

# v=1 is the last flag passed in
kube-apiserver --advertise-address=192.168.64.82 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8443 --service-account-key-file=/var/lib/minikube/certs/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/var/lib/minikube/certs/apiserver.crt --tls-private-key-file=/var/lib/minikube/certs/apiserver.key --v=1

So the flag is being passed in as expected. Could you see if this works for you?

I suspect this will only work on fresh start and not an existing cluster; that issue is being tracked at #8242

[priyawadhwa]

Hey @franck102 are you still seeing this issue?

[jot-hub]

Hi, I experienced the issue with k8s 1.16.3 / minikube 1.12.0. After following this I switched the k8s version to 1.13.10 and minikube api-server could pickup the extra-config for kube-apiserver - i.e. issue didn't happen with k8s 1.13.0.

[priyawadhwa]

Hey @jot-hub I tried with:

• minikube v1.12.2 + k8s version 1.18.3 (Default in minikube)
• minikube v1.12.2 + k8s version 1.16.3

and it worked as expected.

Could you try upgrading your minikube version?

[jot-hub]

Thanks @priyawadhwa for checking!
with the latest versions (1.12.2 and k8s 1.18.3)
minikube start --extra-config=apiserver.v=1 works
minikube start --extra-config=apiserver.oidc-issuer-url="https://joncarl.auth0.com/" does not work - i.e. ps aux |grep oidc returns no results in minikube ssh.

not sure how the parameter makes a difference here

[jot-hub]

update: following sequence of commands worked:

1. minikube delete
2. minikube start --extra-config=apiserver.authorization-mode=RBAC \ --extra-config=apiserver.oidc-issuer-url="https://joncarl.auth0.com/" \ --extra-config=apiserver.oidc-client-id=minikube \ --extra-config=apiserver.oidc-username-claim=http://minikube/email \ --extra-config=apiserver.oidc-groups-claim=http://minikube/groups \ --extra-config=apiserver.oidc-groups-prefix=minikube-
3. minikube ssh
4. ps aux | grep oidc returned kube-apiserver being run with the supplied params

so the issue seems to be only with an existing cluster like suggested earlier in this thread

[tstromberg]

This issue appears to be a duplicate of #8242, do you mind if we move the conversation there?

Ths way we can centralize the content relating to the issue. If you feel that this issue is not in fact a duplicate, please re-open it using /reopen. If you have additional information to share, please add it to the new issue.

Thank you for reporting this!