Warn when /var disk space is limited

[tstromberg]

To replicate:

1. Set Docker to 8GB
2. Gather minikube dependencies: minikube start --download-only
3. Fill the Docker image cache up until docker system df is full. If you need help doing so, I found this works:

seq 32 1 | xargs -n1 -I{} docker pull fedora:{}
seq 8 1 | xargs -n1 -I{} docker pull oraclelinux:{} 
seq 20 1 | xargs -n1 -I{} echo docker pull ubuntu:{}:10
seq 20 1 | xargs -n1 -I{} echo docker pull ubuntu:{}:04

You'll probably see at this point errors, such as no space left on device -- that's a good thing!

4. Run minikube start

Here's what I see

Notice that there is no warning:

😄  minikube v1.12.2 on Darwin 10.15.5
✨  Automatically selected the docker driver. Other choices: hyperkit, vmware, parallels, virtualbox, vmwarefusion
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3892MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

/var is completely full!

% minikube ssh "df -h"
Filesystem      Size  Used Avail Use% Mounted on
overlay         7.9G  7.8G     0 100% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/vda1       7.9G  7.8G     0 100% /var
tmpfs           2.0G  8.5M  2.0G   1% /run
tmpfs           2.0G   12K  2.0G   1% /tmp
/dev/root       390M  390M     0 100% /usr/lib/modules
tmpfs           5.0M     0  5.0M   0% /run/lock

node shows as ready:

NAME       STATUS   ROLES    AGE     VERSION
minikube   Ready    master   3m21s   v1.18.3

Describe shows no disk pressure:

Name:               minikube
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=minikube
                    kubernetes.io/os=linux
                    minikube.k8s.io/commit=be7c19d391302656d27f1f213657d925c4e1cfc2-dirty
                    minikube.k8s.io/name=minikube
                    minikube.k8s.io/updated_at=2020_08_18T12_26_43_0700
                    minikube.k8s.io/version=v1.12.2
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 18 Aug 2020 12:26:40 -0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  minikube
  AcquireTime:     <unset>
  RenewTime:       Tue, 18 Aug 2020 12:29:54 -0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 18 Aug 2020 12:26:55 -0700   Tue, 18 Aug 2020 12:26:36 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 18 Aug 2020 12:26:55 -0700   Tue, 18 Aug 2020 12:26:36 -0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 18 Aug 2020 12:26:55 -0700   Tue, 18 Aug 2020 12:26:36 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Tue, 18 Aug 2020 12:26:55 -0700   Tue, 18 Aug 2020 12:26:55 -0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.17.0.3
  Hostname:    minikube
Capacity:
  cpu:                2
  ephemeral-storage:  8190392Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4035432Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  8190392Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4035432Ki
  pods:               110
System Info:
  Machine ID:                 b95ed2bfb4c74ec9b5b19a9e71395509
  System UUID:                5e5d6d07-9c65-40ac-90e3-2e5e74257084
  Boot ID:                    2a9ce373-3ed6-4829-acc5-3a49f09ddb6f
  Kernel Version:             4.19.76-linuxkit
  OS Image:                   Ubuntu 20.04 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.8
  Kubelet Version:            v1.18.3
  Kube-Proxy Version:         v1.18.3
Non-terminated Pods:          (7 in total)
  Namespace                   Name                                CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-66bff467f8-4vm7b            100m (5%)     0 (0%)      70Mi (1%)        170Mi (4%)     3m15s
  kube-system                 etcd-minikube                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         3m18s
  kube-system                 kube-apiserver-minikube             250m (12%)    0 (0%)      0 (0%)           0 (0%)         3m18s
  kube-system                 kube-controller-manager-minikube    200m (10%)    0 (0%)      0 (0%)           0 (0%)         3m18s
  kube-system                 kube-proxy-k6lnp                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3m15s
  kube-system                 kube-scheduler-minikube             100m (5%)     0 (0%)      0 (0%)           0 (0%)         3m18s
  kube-system                 storage-provisioner                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         3m18s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                650m (32%)  0 (0%)
  memory             70Mi (1%)   170Mi (4%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                   Age    From                  Message
  ----     ------                   ----   ----                  -------
  Normal   Starting                 3m20s  kubelet, minikube     Starting kubelet.
  Normal   NodeHasSufficientMemory  3m20s  kubelet, minikube     Node minikube status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    3m20s  kubelet, minikube     Node minikube status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     3m20s  kubelet, minikube     Node minikube status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  3m19s  kubelet, minikube     Updated Node Allocatable limit across pods
  Warning  readOnlySysFS            3m14s  kube-proxy, minikube  CRI error: /sys is read-only: cannot modify conntrack limits, problems may arise later (If running Docker, see docker issue #24000)
  Normal   Starting                 3m14s  kube-proxy, minikube  Starting kube-proxy.
  Normal   NodeReady                3m9s   kubelet, minikube     Node minikube status is now: NodeReady

It seems that the kubelet is not configured to track /var.

[tstromberg]

After starting minikube, docker system df shows:

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              30                  1                   5.698GB             4.745GB (83%)
Containers          1                   1                   3.16MB              0B (0%)
Local Volumes       1                   1                   1.348GB             0B (0%)
Build Cache         0                   0                   0B                  0B

Note that it doesn't say 100% - that's because minikube is running. Once you stop minikube it will say 100%:

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              30                  0                   5.698GB             5.698GB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

[tstromberg]

FWIW, I've helped 3 different people this last week with this issue .. hence the priority of soon.

[tstromberg]

Also, this is unlikely to be specific to Docker or KIC, but it's almost certainly exacerbated by the fact that /var shares disk space with other images.

[priyawadhwa]

minikube start fails for me when /var is filled:

$ minikube start --driver docker --memory 2000
😄  minikube v1.12.3 on Darwin 10.15.6
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=2000MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
❌  Unable to load cached images: loading cached images: transferring cached image: sudo test -d /var/lib/minikube/images && sudo scp -t /var/lib/minikube/images && sudo touch -d "2020-07-07 22:39:55.97626885 -0400" /var/lib/minikube/images/pause_3.2: Process exited with status 1
output: scp: /var/lib/minikube/images/pause_3.2: No space left on device


💣  Failed to update cluster: updating control plane: copy: mkdir: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube: Process exited with status 1
stdout:

stderr:
mkdir: cannot create directory '/etc/systemd/system/kubelet.service.d': No space left on device


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

It would be nice if we could give the user a nicer error message when the "No space left on device" error comes up.

I think we should also warn them ASAP, which would be as soon as the container is created. This means we'll have to go for the "minikube ssh" approach since k8s won't be up yet.

[medyagh]

I think we should also warn them ASAP, which would be as soon as the container is created. This mean

thats a good idea, I agree

[prakash962930]

X Docker is nearly out of disk space, which may cause deployments to fail! (98% of capacity). You can pass '--force' to skip this check.

• Suggestion:

  Try one or more of the following to free up space on the device:

  1. Run "docker system prune" to remove unused Docker data (optionally with "-a")
  2. Increase the storage allocated to Docker for Desktop by clicking on:
     Docker icon > Preferences > Resources > Disk Image Size
  3. Run "minikube ssh -- docker system prune" if using the Docker container runtime

• Related issue: Warn when /var disk space is limited #9024

• Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...

• Configuring Calico (Container Networking Interface) ...

• Verifying Kubernetes components...

  • Using image gcr.io/k8s-minikube/storage-provisioner:v5

• Enabled addons: storage-provisioner, default-storageclass

• Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

[1dropaflame]

Run "docker system prune" to remove unused Docker data (optionally with "-a")

Does not work. I get ZERO bytes reclaimed.

ropaflame@1dropaflame:~$ docker system prune

WARNING! This will remove:

  - all stopped containers

  - all networks not used by at least one container

  - all dangling images

  - all dangling build cache

Are you sure you want to continue? [y/N] y

Total reclaimed space: 0B

Update:

I had to do a

minikube ssh docker system prune
and then it worked

[vanityjane]

python hello