auto-pause addon: automatically pause Kubernetes when not in use #10427

medyagh · 2021-02-09T22:19:02Z

Quick Demo

start minikube and Enable auto-pause addon

medya@~/workspace/minikube (auto-pause) $ mk addons enable auto-pause
    ▪ Using image gcr.io/haproxy:2.3.5
🌟  The 'auto-pause' addon is enabled

checkout the pods

medya@~/workspace/minikube (auto-pause) $ kc get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
auto-pause    auto-pause-proxy-dc6979c79-c2qz2   1/1     Running   0          7s
kube-system   coredns-74ff55c5b-7ks55            1/1     Running   0          6m10s
kube-system   etcd-minikube                      1/1     Running   0          6m25s
kube-system   kube-apiserver-minikube            1/1     Running   0          6m25s
kube-system   kube-controller-manager-minikube   1/1     Running   0          6m25s
kube-system   kube-proxy-578jp                   1/1     Running   0          6m10s
kube-system   kube-scheduler-minikube            1/1     Running   0          6m25s
kube-system   storage-provisioner                1/1     Running   1          6m24s

check minukube status

medya@~/workspace/minikube (auto-pause) $ mk status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
timeToStop: Nonexistent

wait X minutes

medya@~/workspace/minikube (auto-pause) $ date
Sun Feb 21 22:32:11 PST 2021
medya@~/workspace/minikube (auto-pause) $ mk status
minikube
type: Control Plane
host: Running
kubelet: Stopped
apiserver: Paused
kubeconfig: Configured
timeToStop: Nonexistent

verify the containers paused

medya@~/workspace/minikube (auto-pause) $ mk ssh docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS                  PORTS                     NAMES
f8315dcb1bff   307c153b6ca1           "docker-entrypoint.s…"   About a minute ago   Up About a minute                                 k8s_auto-pause_auto-pause-proxy-dc6979c79-c2qz2_auto-pause_0169c9b9-a40c-475c-9b1a-dd14a64bd2ee_0
2f21b763c367   k8s.gcr.io/pause:3.2   "/pause"                 About a minute ago   Up About a minute       0.0.0.0:32443->6443/tcp   k8s_POD_auto-pause-proxy-dc6979c79-c2qz2_auto-pause_0169c9b9-a40c-475c-9b1a-dd14a64bd2ee_0
7201e6a0ec37   85069258b98a           "/storage-provisioner"   7 minutes ago        Up 7 minutes (Paused)                             k8s_storage-provisioner_storage-provisioner_kube-system_42bc2995-c5bc-4d0b-8baf-faea97e704e3_1
2f8720ae7c87   bfe3a36ebd25           "/coredns -conf /etc…"   7 minutes ago        Up 7 minutes (Paused)                             k8s_coredns_coredns-74ff55c5b-7ks55_kube-system_b934043c-7078-4814-9ec8-f81c7a60f18f_0
5b36e5f93cc6   43154ddb57a8           "/usr/local/bin/kube…"   7 minutes ago        Up 7 minutes (Paused)                             k8s_kube-proxy_kube-proxy-578jp_kube-system_2bf5b7a7-a764-412a-a41d-6999d64ded5b_0
8a2d2dbd4b79   k8s.gcr.io/pause:3.2   "/pause"                 7 minutes ago        Up 7 minutes (Paused)                             k8s_POD_coredns-74ff55c5b-7ks55_kube-system_b934043c-7078-4814-9ec8-f81c7a60f18f_0
0619b7329bfd   k8s.gcr.io/pause:3.2   "/pause"                 7 minutes ago        Up 7 minutes (Paused)                             k8s_POD_storage-provisioner_kube-system_42bc2995-c5bc-4d0b-8baf-faea97e704e3_0
62ffdb351c54   k8s.gcr.io/pause:3.2   "/pause"                 7 minutes ago        Up 7 minutes (Paused)                             k8s_POD_kube-proxy-578jp_kube-system_2bf5b7a7-a764-412a-a41d-6999d64ded5b_0
05a686b38038   ed2c44fbdd78           "kube-scheduler --au…"   8 minutes ago        Up 8 minutes (Paused)                             k8s_kube-scheduler_kube-scheduler-minikube_kube-system_6b4a0ee8b3d15a1c2e47c15d32e6eb0d_0
5a6bd92f4452   a27166429d98           "kube-controller-man…"   8 minutes ago        Up 8 minutes (Paused)                             k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_57b8c22dbe6410e4bd36cf14b0f8bdc7_0
e0ac167f34ad   0369cf4303ff           "etcd --advertise-cl…"   8 minutes ago        Up 8 minutes (Paused)                             k8s_etcd_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_0
d040db588c01   k8s.gcr.io/pause:3.2   "/pause"                 8 minutes ago        Up 8 minutes (Paused)                             k8s_POD_kube-scheduler-minikube_kube-system_6b4a0ee8b3d15a1c2e47c15d32e6eb0d_0
cf4cb6d6f59c   k8s.gcr.io/pause:3.2   "/pause"                 8 minutes ago        Up 8 minutes (Paused)                             k8s_POD_kube-controller-manager-minikube_kube-system_57b8c22dbe6410e4bd36cf14b0f8bdc7_0
00c676197922   a8c2fdb8bf76           "kube-apiserver --ad…"   8 minutes ago        Up 8 minutes (Paused)                             k8s_kube-apiserver_kube-apiserver-minikube_kube-system_c767dbeb9ddd2d01964c2fc02c621c4e_0
0fe8483325e8   k8s.gcr.io/pause:3.2   "/pause"                 8 minutes ago        Up 8 minutes (Paused)                             k8s_POD_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_0
c808cd05f8b6   k8s.gcr.io/pause:3.2   "/pause"                 8 minutes ago        Up 8 minutes (Paused)                             k8s_POD_kube-apiserver-minikube_kube-system_c767dbeb9ddd2d01964c2fc02c621c4e_0
medya@~/workspace/minikube (auto-pause) $ 
medya@~/workspace/minikube (auto-pause) $ code ~/.k
medya@~/workspace/minikube (auto-pause) $ time KUBECONFIG=/Users/medya/.kube/config-ap kc get pods -A 
Error: unknown flag: --alsologtostde
See 'kubectl get --help' for usage.

real    0m0.173s
user    0m0.058s
sys     0m0.036s

invoke a kubectl get pods

 kc get pods -A 
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
auto-pause    auto-pause-proxy-dc6979c79-c2qz2   1/1     Running   0          3m32s
kube-system   coredns-74ff55c5b-7ks55            1/1     Running   0          9m35s
kube-system   etcd-minikube                      1/1     Running   0          9m50s
kube-system   kube-apiserver-minikube            1/1     Running   0          9m50s
kube-system   kube-controller-manager-minikube   1/1     Running   0          9m50s
kube-system   kube-proxy-578jp                   1/1     Running   0          9m35s
kube-system   kube-scheduler-minikube            1/1     Running   0          9m50s
kube-system   storage-provisioner                1/1     Running   1          9m49s

real    0m4.139s
user    0m0.137s
sys     0m0.218s

verify it unpaused minikube

medya@~/workspace/minikube (auto-pause) $ mk status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
timeToStop: Nonexistent

verify containers


medya@~/workspace/minikube (auto-pause) $ mk ssh docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                     NAMES
f8315dcb1bff   307c153b6ca1           "docker-entrypoint.s…"   3 minutes ago    Up 3 minutes                              k8s_auto-pause_auto-pause-proxy-dc6979c79-c2qz2_auto-pause_0169c9b9-a40c-475c-9b1a-dd14a64bd2ee_0
2f21b763c367   k8s.gcr.io/pause:3.2   "/pause"                 3 minutes ago    Up 3 minutes    0.0.0.0:32443->6443/tcp   k8s_POD_auto-pause-proxy-dc6979c79-c2qz2_auto-pause_0169c9b9-a40c-475c-9b1a-dd14a64bd2ee_0
7201e6a0ec37   85069258b98a           "/storage-provisioner"   9 minutes ago    Up 9 minutes                              k8s_storage-provisioner_storage-provisioner_kube-system_42bc2995-c5bc-4d0b-8baf-faea97e704e3_1
2f8720ae7c87   bfe3a36ebd25           "/coredns -conf /etc…"   9 minutes ago    Up 9 minutes                              k8s_coredns_coredns-74ff55c5b-7ks55_kube-system_b934043c-7078-4814-9ec8-f81c7a60f18f_0
5b36e5f93cc6   43154ddb57a8           "/usr/local/bin/kube…"   9 minutes ago    Up 9 minutes                              k8s_kube-proxy_kube-proxy-578jp_kube-system_2bf5b7a7-a764-412a-a41d-6999d64ded5b_0
8a2d2dbd4b79   k8s.gcr.io/pause:3.2   "/pause"                 9 minutes ago    Up 9 minutes                              k8s_POD_coredns-74ff55c5b-7ks55_kube-system_b934043c-7078-4814-9ec8-f81c7a60f18f_0
0619b7329bfd   k8s.gcr.io/pause:3.2   "/pause"                 9 minutes ago    Up 9 minutes                              k8s_POD_storage-provisioner_kube-system_42bc2995-c5bc-4d0b-8baf-faea97e704e3_0
62ffdb351c54   k8s.gcr.io/pause:3.2   "/pause"                 9 minutes ago    Up 9 minutes                              k8s_POD_kube-proxy-578jp_kube-system_2bf5b7a7-a764-412a-a41d-6999d64ded5b_0
05a686b38038   ed2c44fbdd78           "kube-scheduler --au…"   10 minutes ago   Up 10 minutes                             k8s_kube-scheduler_kube-scheduler-minikube_kube-system_6b4a0ee8b3d15a1c2e47c15d32e6eb0d_0
5a6bd92f4452   a27166429d98           "kube-controller-man…"   10 minutes ago   Up 10 minutes                             k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_57b8c22dbe6410e4bd36cf14b0f8bdc7_0
e0ac167f34ad   0369cf4303ff           "etcd --advertise-cl…"   10 minutes ago   Up 10 minutes                             k8s_etcd_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_0
d040db588c01   k8s.gcr.io/pause:3.2   "/pause"                 10 minutes ago   Up 10 minutes                             k8s_POD_kube-scheduler-minikube_kube-system_6b4a0ee8b3d15a1c2e47c15d32e6eb0d_0
cf4cb6d6f59c   k8s.gcr.io/pause:3.2   "/pause"                 10 minutes ago   Up 10 minutes                             k8s_POD_kube-controller-manager-minikube_kube-system_57b8c22dbe6410e4bd36cf14b0f8bdc7_0
00c676197922   a8c2fdb8bf76           "kube-apiserver --ad…"   10 minutes ago   Up 10 minutes                             k8s_kube-apiserver_kube-apiserver-minikube_kube-system_c767dbeb9ddd2d01964c2fc02c621c4e_0
0fe8483325e8   k8s.gcr.io/pause:3.2   "/pause"                 10 minutes ago   Up 10 minutes                             k8s_POD_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_0
c808cd05f8b6   k8s.gcr.io/pause:3.2   "/pause"                 10 minutes ago   Up 10 minutes                             k8s_POD_kube-apiserver-minikube_kube-system_c767dbeb9ddd2d01964c2fc02c621c4e_0

TODO:

ensure a paused cluster can be deleted. ( I saw a flake case that delete was stuck)

Follow up PRs coming after initial prototype.

support non docker runtimes
add flag alias to enable auto-pause (mk start --auto-pause)
make time to pause configurable
support arm64
ensure when disabling auto-pause it unpauses minikube

medyagh · 2021-02-20T21:25:45Z

ok-to-build-image

minikube-bot · 2021-02-20T21:35:47Z

Hi @medyagh, building a new kicbase image failed, please try again.

medyagh · 2021-02-20T21:53:54Z

ok-to-build-image

minikube-bot · 2021-02-20T22:08:07Z

Hi @medyagh, building a new kicbase image failed, please try again.

medyagh · 2021-02-20T22:26:36Z

ok-to-build-image

minikube-bot · 2021-02-20T22:35:24Z

Hi @medyagh, building a new kicbase image failed, please try again.

medyagh · 2021-02-20T22:50:14Z

ok-to-build-image

minikube-bot · 2021-02-20T23:01:57Z

Hi @medyagh,

A new kicbase image is available, please update your PR with the new tag and SHA.
In pkg/drivers/kic/types.go:

// Version is the current version of kic
Version = "v0.0.17-1613861585-10427"
// SHA of the kic base image
baseImageSHA = "c0827e1b58f31cbfeb871af9c7656e5822bd97dd0011b9edd45ddd1ee9ef9dfd"
// The name of the GCR kicbase repository
gcrRepo = "gcr.io/k8s-minikube/kicbase-builds"
// The name of the Dockerhub kicbase repository
dockerhubRepo = "kicbase/build"

Then run make generate-docs to update our documentation to reference the new image.

Alternatively, run the following command and commit the changes:

 sed 's|Version = .*|Version = "v0.0.17-1613861585-10427"|;s|baseImageSHA = .*|baseImageSHA = "c0827e1b58f31cbfeb871af9c7656e5822bd97dd0011b9edd45ddd1ee9ef9dfd"|;s|gcrRepo = .*|gcrRepo = "gcr.io/k8s-minikube/kicbase-builds"|;s|dockerhubRepo = .*|dockerhubRepo = "kicbase/build"|' pkg/drivers/kic/types.go > new-types.go; mv new-types.go pkg/drivers/kic/types.go; make generate-docs;

priyawadhwa

LGTM! Just a couple nits and small suggestions.

deploy/addons/auto-pause/auto-pause.yaml.tmpl

pkg/addons/addons.go

cmd/auto-pause/auto-pause.go

Makefile

cmd/auto-pause/auto-pause.go

deploy/addons/auto-pause/auto-pause.service

medyagh · 2021-02-24T21:17:42Z

ok-to-build-image

minikube-bot · 2021-02-24T21:26:11Z

Hi @medyagh, your kicbase info is out of date. Please rebase.

medyagh · 2021-02-24T21:29:27Z

ok-to-build-image

minikube-bot · 2021-02-24T21:44:52Z

Hi @medyagh,

A new kicbase image is available, please update your PR with the new tag and SHA.
In pkg/drivers/kic/types.go:

// Version is the current version of kic
Version = "v0.0.17-1614202509-10427"
// SHA of the kic base image
baseImageSHA = "93f2448d272ebad3d564c04474cafe03bb7440149c241d1d010b2e90e14da213"
// The name of the GCR kicbase repository
gcrRepo = "gcr.io/k8s-minikube/kicbase-builds"
// The name of the Dockerhub kicbase repository
dockerhubRepo = "kicbase/build"

Then run make generate-docs to update our documentation to reference the new image.

Alternatively, run the following command and commit the changes:

 sed 's|Version = .*|Version = "v0.0.17-1614202509-10427"|;s|baseImageSHA = .*|baseImageSHA = "93f2448d272ebad3d564c04474cafe03bb7440149c241d1d010b2e90e14da213"|;s|gcrRepo = .*|gcrRepo = "gcr.io/k8s-minikube/kicbase-builds"|;s|dockerhubRepo = .*|dockerhubRepo = "kicbase/build"|' pkg/drivers/kic/types.go > new-types.go; mv new-types.go pkg/drivers/kic/types.go; make generate-docs;

medyagh · 2021-02-24T21:51:25Z

/ok-to-test

k8s-ci-robot · 2021-02-24T22:00:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 9, 2021

k8s-ci-robot requested review from priyawadhwa and sharifelgamal February 9, 2021 22:19

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 9, 2021

medyagh force-pushed the auto-pause branch from 5359a6c to 2c8b4b4 Compare February 20, 2021 21:24

medyagh mentioned this pull request Feb 20, 2021

minikube kic-auto build should provide kic image build error #10543

Closed

medyagh force-pushed the auto-pause branch from 90d1b6d to 020001b Compare February 21, 2021 20:38

medyagh mentioned this pull request Feb 22, 2021

refactor: ClusterConfig does not store the addons map #10564

Closed

medyagh changed the title ~~wip: auto pause~~ auto-pause feature: automatically pause kubernetes when it is not in use Feb 22, 2021

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 22, 2021

medyagh force-pushed the auto-pause branch from 16df330 to 0d3c949 Compare February 22, 2021 20:41

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 22, 2021

priyawadhwa reviewed Feb 22, 2021

View reviewed changes

deploy/addons/auto-pause/auto-pause.yaml.tmpl Outdated Show resolved Hide resolved

pkg/addons/addons.go Outdated Show resolved Hide resolved

pkg/addons/addons.go Outdated Show resolved Hide resolved

cmd/auto-pause/auto-pause.go Show resolved Hide resolved

spowelljr reviewed Feb 22, 2021

View reviewed changes

Makefile Show resolved Hide resolved