kubelet: The generated config is not loaded (imagefs.available not applied)

[maingoh]

Minikube generates a kubelet config file in /var/lib/kubelet/config.yaml with eviction thresholds at 0%.
However it doesn't seem to be loaded by kubelet.

The exact command to reproduce the issue:

$ minikube start --vm-driver=none --extra-config=kubelet.v=20
😄  minikube v1.3.1 on Ubuntu 18.04
🤹  Running on localhost (CPUs=16, Memory=32176MB, Disk=436619MB) ...
ℹ️   OS release is Ubuntu 18.04.1 LTS
🐳  Preparing Kubernetes v1.15.2 on Docker 19.03.2 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
💾  Downloading kubeadm v1.15.2
💾  Downloading kubelet v1.15.2
🚜  Pulling images ...
🚀  Launching Kubernetes ... 
🤹  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
👉  https://minikube.sigs.k8s.io/docs/reference/drivers/none/

⌛  Waiting for: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"

$ sudo journalctl -u kubelet --since=today | grep imagefs.available
Sep 12 15:34:40 maingoh kubelet[30706]: I0912 15:34:40.607950   30706 server.go:271] KubeletConfiguration: config.KubeletConfiguration{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, StaticPodPath:"/etc/kubernetes/manifests", SyncFrequency:v1.Duration{Duration:60000000000}, FileCheckFrequency:v1.Duration{Duration:20000000000}, HTTPCheckFrequency:v1.Duration{Duration:20000000000}, StaticPodURL:"", StaticPodURLHeader:map[string][]string(nil), Address:"0.0.0.0", Port:10250, ReadOnlyPort:10255, TLSCertFile:"/var/lib/kubelet/pki/kubelet.crt", TLSPrivateKeyFile:"/var/lib/kubelet/pki/kubelet.key", TLSCipherSuites:[]string(nil), TLSMinVersion:"", RotateCertificates:false, ServerTLSBootstrap:false, Authentication:config.KubeletAuthentication{X509:config.KubeletX509Authentication{ClientCAFile:"/var/lib/minikube/certs/ca.crt"}, Webhook:config.KubeletWebhookAuthentication{Enabled:false, CacheTTL:v1.Duration{Duration:120000000000}}, Anonymous:config.KubeletAnonymousAuthentication{Enabled:true}}, Authorization:config.KubeletAuthorization{Mode:"Webhook", Webhook:config.KubeletWebhookAuthorization{CacheAuthorizedTTL:v1.Duration{Duration:300000000000}, CacheUnauthorizedTTL:v1.Duration{Duration:30000000000}}}, RegistryPullQPS:5, RegistryBurst:10, EventRecordQPS:5, EventBurst:10, EnableDebuggingHandlers:true, EnableContentionProfiling:false, HealthzPort:10248, HealthzBindAddress:"127.0.0.1", OOMScoreAdj:-999, ClusterDomain:"cluster.local", ClusterDNS:[]string{"10.96.0.10"}, StreamingConnectionIdleTimeout:v1.Duration{Duration:14400000000000}, NodeStatusUpdateFrequency:v1.Duration{Duration:10000000000}, NodeStatusReportFrequency:v1.Duration{Duration:60000000000}, NodeLeaseDurationSeconds:40, ImageMinimumGCAge:v1.Duration{Duration:120000000000}, ImageGCHighThresholdPercent:85, ImageGCLowThresholdPercent:80, VolumeStatsAggPeriod:v1.Duration{Duration:60000000000}, KubeletCgroups:"", SystemCgroups:"", CgroupRoot:"", CgroupsPerQOS:true, CgroupDriver:"cgroupfs", CPUManagerPolicy:"none", CPUManagerReconcilePeriod:v1.Duration{Duration:10000000000}, QOSReserved:map[string]string(nil), RuntimeRequestTimeout:v1.Duration{Duration:120000000000}, HairpinMode:"promiscuous-bridge", MaxPods:110, PodCIDR:"", PodPidsLimit:-1, ResolverConfig:"/run/systemd/resolve/resolv.conf", CPUCFSQuota:true, CPUCFSQuotaPeriod:v1.Duration{Duration:100000000}, MaxOpenFiles:1000000, ContentType:"application/vnd.kubernetes.protobuf", KubeAPIQPS:5, KubeAPIBurst:10, SerializeImagePulls:true, EvictionHard:map[string]string{"imagefs.available":"15%", "memory.available":"100Mi", "nodefs.available":"10%", "nodefs.inodesFree":"5%"}, EvictionSoft:map[string]string(nil), EvictionSoftGracePeriod:map[string]string(nil), EvictionPressureTransitionPeriod:v1.Duration{Duration:300000000000}, EvictionMaxPodGracePeriod:0, EvictionMinimumReclaim:map[string]string(nil), PodsPerCore:0, EnableControllerAttachDetach:true, ProtectKernelDefaults:false, MakeIPTablesUtilChains:true, IPTablesMasqueradeBit:14, IPTablesDropBit:15, FeatureGates:map[string]bool(nil), FailSwapOn:false, ContainerLogMaxSize:"10Mi", ContainerLogMaxFiles:5, ConfigMapAndSecretChangeDetectionStrategy:"Watch", SystemReserved:map[string]string(nil), KubeReserved:map[string]string(nil), SystemReservedCgroup:"", KubeReservedCgroup:"", EnforceNodeAllocatable:[]string{"pods"}}

Those are the default thresholds.

The operating system version: Ubuntu 18.04 / Minikube 1.3.1

[thomas-riccardi]

This issue breaks the configuration that disables the default image pruning on disk space pressure. With vm=none it is quite severe: it deletes images on the host docker (with a sub-optimal priority: it only cares about images used in minikube pods, which is not the reality in a developer machine).

Would it be possible to increase priority?

Our current workaround is to call minikube with --extra-config=kubelet.config=/var/lib/kubelet/config.yaml, but it should probably be done by default: with raw kubeadm this file option is added via the kubelet drop-in file for systemd.
See also this more generic doc about this config file generation.

[sharifelgamal]

Yep, this is definitely an issue. We'll move it up the stack and get to it soon.

[nanikjava]

Yep, this is definitely an issue. We'll move it up the stack and get to it soon.

@sharifelgamal is this being worked on ?, if not then would be interested to take this but will need bit more context on how to reproduce the issue.
Cheers

[thomas-riccardi]

@nanikjava caafc66 explicitly added configuration in kubeadm to disable kubelet disk eviction (which is the feature we need).
As explained in the previous comments, this configuration is correctly written to disk by kubeadm during minikube start (in /var/lib/kubelet/config.yaml), but it is not used by kubelet.

The error is there: kubelet should be configured to read that config. See the documentation I linked in my previous comment.

[nanikjava]

Will test this out. Thanks @thomas-riccardi

[nanikjava]

/assign @nanikjava

[nanikjava]

Tried running minikube using VirtualBox and run it with the following command

minikube start

Can see inside the VM kubelet ran with the following command

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2019-10-19 10:09:57 UTC; 5min ago
Docs: http://kubernetes.io/docs/
Main PID: 3616 (kubelet)
Tasks: 17 (limit: 2175)
CGroup: /system.slice/kubelet.service
└─3616 /var/lib/minikube/binaries/v1.16.1/kubelet --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.99.205 --pod-manifest-path=/etc/kubernetes/manifests

...and tried running with the following command

minikube start --extra-config=kubelet.config=/var/lib/kubelet/config.yaml

can see the config.yaml is attached to the command line

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2019-10-19 10:17:43 UTC; 1min 31s ago
Docs: http://kubernetes.io/docs/
Main PID: 8128 (kubelet)
Tasks: 18 (limit: 2175)
CGroup: /system.slice/kubelet.service
└─8128 /var/lib/minikube/binaries/v1.16.1/kubelet --config=/var/lib/kubelet/config.yaml --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.99.205 --pod-manifest-path=/etc/kubernetes/manifests

is this the problem where the config.yaml is not 'attach' as part of the command line to run kubelet ?

[thomas-riccardi]

@nanikjava yes, the kubelet --config=/var/lib/kubelet/config.yaml is probably the exepected behavior.

[nanikjava]

thanks @thomas-riccardi will put a fix for this.

[nanikjava]

@tstromberg @sharifelgamal PR submitted #5697

[nanikjava]

@tstromberg can we close this issue ?. Thanks

[tstromberg]

/close

…

On Thu, Nov 7, 2019, 1:12 PM Nanik ***@***.***> wrote: @tstromberg <https://github.com/tstromberg> can we close this issue ?. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5329?email_source=notifications&email_token=AAAYYMC3VUQAK2JIVKB5ODTQSSAC7A5CNFSM4IWEG7Y2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDN2PQA#issuecomment-551266240>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAYYMHNJBKUWUE4WEFB2VTQSSAC7ANCNFSM4IWEG7YQ> .

[k8s-ci-robot]

@tstromberg: Closing this issue.

In response to this:

/close

On Thu, Nov 7, 2019, 1:12 PM Nanik [email protected] wrote:

@tstromberg https://github.com/tstromberg can we close this issue ?.
Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#5329,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAYYMHNJBKUWUE4WEFB2VTQSSAC7ANCNFSM4IWEG7YQ
.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.