Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running out of disk should set cluster status to 507 (Insufficient Storage) #9025

Closed
tstromberg opened this issue Aug 18, 2020 · 1 comment · Fixed by #9034
Closed

Running out of disk should set cluster status to 507 (Insufficient Storage) #9025

tstromberg opened this issue Aug 18, 2020 · 1 comment · Fixed by #9034
Assignees
Labels
kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@tstromberg
Copy link
Contributor

Related to #9024

If /var is full, it should be addressed in minikube status --layout=cluster -o json.

docker@minikube:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         7.9G  7.8G     0 100% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/vda1       7.9G  7.8G     0 100% /var
tmpfs           2.0G  8.5M  2.0G   1% /run
tmpfs           2.0G   12K  2.0G   1% /tmp
/dev/root       390M  390M     0 100% /usr/lib/modules
tmpfs           5.0M     0  5.0M   0% /run/lock

Everything looks OK, but it isn't:

% minikube status --layout=cluster -o json | jq                                                                                                                                  {
  "Name": "minikube",
  "StatusCode": 200,
  "StatusName": "OK",
  "Step": "Done",
  "StepDetail": "🏄  Done! kubectl is now configured to use \"minikube\"",
  "BinaryVersion": "v1.12.2",
  "Components": {
    "kubeconfig": {
      "Name": "kubeconfig",
      "StatusCode": 200,
      "StatusName": ""
    }
  },
  "Nodes": [
    {
      "Name": "minikube",
      "StatusCode": 200,
      "StatusName": "OK",
      "Components": {
        "apiserver": {
          "Name": "apiserver",
          "StatusCode": 200,
          "StatusName": "OK"
        },
        "kubelet": {
          "Name": "kubelet",
          "StatusCode": 200,
          "StatusName": "OK"
        }
      }
    }
  ]
}

Ideally, I would like to see both the node and cluster state populated with something like this when completely full:

  "StatusCode": 507,
  "StatusName": "INSUFFICIENT_STORAGE",
  "StatusDetail": "/var is out of disk space"

or if it's nearly out of space:

  "StatusCode": 203,
  "StatusName": "WARNING",
  "StatusDetail": "/var is almost of disk space (90%)"

Doing so via kubelet DiskPressure check would be ideal, but if need be, we can use minikube ssh to simulate this check initially.

@tstromberg tstromberg changed the title Running out of disk should affect the cluster/node Status Running out of disk should set status to 507 (Insufficient Storage) Aug 18, 2020
@tstromberg tstromberg changed the title Running out of disk should set status to 507 (Insufficient Storage) Running out of disk should set cluster status to 507 (Insufficient Storage) Aug 18, 2020
@tstromberg tstromberg added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Aug 18, 2020
@tstromberg tstromberg added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 20, 2020
@afbjorklund
Copy link
Collaborator

Related to #3574

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants