Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

faster containerd start by preloading images #7793

Merged
merged 14 commits into from
Apr 20, 2020

Conversation

medyagh
Copy link
Member

@medyagh medyagh commented Apr 20, 2020

note1

the times in this PR should be interprteded Relative to each other, since I run it on a very old computer.
on modern computer all of the the times bellow are -20 seconds.

After this PR

make containerd 33 start seconds faster !

Revolutionary finding !

containerd is 22 seconds faster than docker-runtime (tested both on docker driver see bellow)
this makes docker driver with containerd Fastest Possible local kubernetes option
which makes sense, since our docker service binds to containerd service.
but containerd only needs 1 service (we dont need run both docker and containerd service)

to run this PR locally see the footnote bellow.

After this PR

$ make && time ./out/minikube start --driver=docker --container-runtime=containerd  --kubernetes-version=v1.17.5
make: 'out/minikube' is up to date.
😄  minikube v1.9.2 on Ubuntu 20.04
✨  Using the docker driver based on user configuration
👍  Kubernetes 1.18.0 is now available. If you would like to upgrade, specify: --kubernetes-version=1.18.0
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
📦  Preparing Kubernetes v1.17.5 on containerd 1.3.3-14-g449e9269 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
🕵️   Verifying Kubernetes Components:
    🔎 verifying node conditions ...
    🔎 verifying api server ...
    🔎 verifying system pods ...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

real    0m41.250s
user    0m2.728s
sys     0m0.900s

Before this PR

(note that the images are cached without cached images it is even much slower)

$ make && time ./out/minikube start --driver=docker --container-runtime=containerd  --kubernetes-version=v1.17.5
make: 'out/minikube' is up to date.
😄  minikube v1.9.2 on Ubuntu 20.04
✨  Using the docker driver based on user configuration
👍  Kubernetes 1.18.0 is now available. If you would like to upgrade, specify: --kubernetes-version=1.18.0
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
📦  Preparing Kubernetes v1.17.5 on containerd 1.3.3-14-g449e9269 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
🕵️   Verifying Kubernetes Components:
    🔎 verifying node conditions ...
    🔎 verifying api server ...
    🔎 verifying system pods ...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

real    1m13.192s
user    0m4.132s
sys     0m1.376s

Docker runtime with same version of kubernetes (on my machine) is 20 seconds slower than containerd

$ make && time ./out/minikube start --driver=docker --container-runtime=docker  --kubernetes-version=v1.17.5

real    1m1.998s
user    0m3.634s
sys     0m1.502s

Footnote1

for this PR, I generated the preload tar using 'make upload-preloaded-images-tar' and manually put it in '~/.minikube/cache/preloaded-tarball/'

once this PR is merged, the generating script should be uploading containerd tarballs as well.

TODOs:

added an investigation that @priyawadhwa could provide good insight on this
#7804

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 20, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 20, 2020
@medyagh medyagh changed the title Preload containerd add Preload feature for containerd Apr 20, 2020
@medyagh medyagh changed the title add Preload feature for containerd add preload feature for containerd Apr 20, 2020
@medyagh
Copy link
Member Author

medyagh commented Apr 20, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Apr 20, 2020
@kubernetes kubernetes deleted a comment from minikube-pr-bot Apr 20, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [62.135730829 62.416243263 62.702258807]
Average time for minikube: 62.418077632999996

Times for Minikube (PR 7793): [65.135420829 67.507113578 61.47996308100001]
Average time for Minikube (PR 7793): 64.70749916266665

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7793) |
+--------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian    |  0.059006 |           0.057404 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.019398 |           0.018774 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003110 |           0.003097 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.483693 |          41.498557 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 19.706259 |          21.105672 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  0.013968 |           0.004473 |
| Components:                    |           |                    |
|   - verifying node conditions  |  0.001468 |           0.001361 |
| ...                            |           |                    |
|   - verifying api server ...   |  0.105210 |           0.078402 |
|   - verifying system pods ...  |  1.554565 |           1.162745 |
| * Enabled addons:              |  0.392927 |           0.703953 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.075497 |           0.068727 |
| configured to use "minikube"   |           |                    |
|                                |  0.002977 |           0.004335 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.199031554 26.499175707 26.317391562000005]
Average time for minikube: 26.338532941

Times for Minikube (PR 7793): [24.985659690000002 24.632839739000005 25.041115957000002]
Average time for Minikube (PR 7793): 24.886538462000004

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7793) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian            |  0.071167 |           0.074287 |
|                                   9.11 |           |                    |
| * Using the docker driver              |  0.002411 |           0.002292 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.056002 |           0.058638 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.375606 |           7.284812 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000198 |           0.000177 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.573647 |          16.522095 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.001373 |           0.001578 |
| Components:                            |           |                    |
|   - verifying node conditions          |  0.003912 |           0.003248 |
| ...                                    |           |                    |
|   - verifying api server ...           |  0.083782 |           0.085353 |
|   - verifying system pods ...          |  1.002393 |           0.673696 |
| * Enabled addons:                      |  0.098953 |           0.117502 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.064922 |           0.059534 |
| configured to use "minikube"           |           |                    |
|                                        |  0.004167 |           0.003326 |
+----------------------------------------+-----------+--------------------+

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [64.145800052 64.95187995099998 64.715996096]
Average time for minikube: 64.60455869966667

Times for Minikube (PR 7793): [65.22681922 63.946924963 65.72724363500001]
Average time for Minikube (PR 7793): 64.96699593933333

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7793) |
+--------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian    |  0.054350 |           0.056940 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.017420 |           0.018714 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.002908 |           0.003274 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.104636 |          41.114218 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 21.546285 |          21.867398 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  0.000853 |           0.001543 |
| Components:                    |           |                    |
|   - verifying node conditions  |  0.001643 |           0.000864 |
| ...                            |           |                    |
|   - verifying api server ...   |  0.077158 |           0.081136 |
|   - verifying system pods ...  |  1.174339 |           1.333362 |
| * Enabled addons:              |  0.547422 |           0.402677 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.074670 |           0.080508 |
| configured to use "minikube"   |           |                    |
|                                |  0.002874 |           0.006361 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [25.729839854999994 25.448882594999997 26.210690158000002]
Average time for minikube: 25.796470869333334

Times for Minikube (PR 7793): [25.352620146 26.924615479 25.301310698]
Average time for Minikube (PR 7793): 25.859515441000003

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7793) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian            |  0.069673 |           0.070628 |
|                                   9.11 |           |                    |
| * Using the docker driver              |  0.002459 |           0.002768 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.055849 |           0.055727 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.316854 |           7.327334 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000146 |           0.000148 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.170428 |          17.035870 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.001306 |           0.002331 |
| Components:                            |           |                    |
|   - verifying node conditions          |  0.004275 |           0.002324 |
| ...                                    |           |                    |
|   - verifying api server ...           |  0.080291 |           0.058538 |
|   - verifying system pods ...          |  1.018897 |           1.087071 |
| * Enabled addons:                      |  0.002208 |           0.148225 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.071297 |           0.062221 |
| configured to use "minikube"           |           |                    |
|                                        |  0.002786 |           0.006330 |
+----------------------------------------+-----------+--------------------+

Copy link

@priyawadhwa priyawadhwa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this, it'll be really useful for containerd users. Left a few comments.

hack/preload-images/generate.go Outdated Show resolved Hide resolved
hack/preload-images/generate.go Outdated Show resolved Hide resolved
pkg/drivers/kic/kic.go Outdated Show resolved Hide resolved
pkg/minikube/cruntime/containerd.go Outdated Show resolved Hide resolved
pkg/minikube/machine/cache_images.go Outdated Show resolved Hide resolved
@codecov-io
Copy link

codecov-io commented Apr 20, 2020

Codecov Report

Merging #7793 into master will decrease coverage by 0.28%.
The diff coverage is 5.19%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7793      +/-   ##
==========================================
- Coverage   35.87%   35.59%   -0.29%     
==========================================
  Files         148      148              
  Lines        9201     9286      +85     
==========================================
+ Hits         3301     3305       +4     
- Misses       5504     5585      +81     
  Partials      396      396              
Impacted Files Coverage Δ
pkg/minikube/cruntime/cruntime.go 58.06% <0.00%> (-6.23%) ⬇️
pkg/minikube/download/preload.go 0.00% <0.00%> (ø)
pkg/minikube/machine/cache_images.go 0.00% <0.00%> (ø)
pkg/minikube/cruntime/containerd.go 28.65% <5.88%> (-14.62%) ⬇️
cmd/minikube/cmd/completion.go 0.00% <0.00%> (ø)

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [60.94407114799999 64.109315799 66.241905156]
Average time for minikube: 63.76509736766667

Times for Minikube (PR 7793): [63.158488817000006 63.36685312300001 66.93133483700001]
Average time for Minikube (PR 7793): 64.48555892566667

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7793) |
+--------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian    |  0.056102 |           0.056800 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.018707 |           0.017630 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.002753 |           0.002859 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 39.947795 |          39.772634 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 21.682839 |          22.381107 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  0.007361 |           0.003514 |
| Components:                    |           |                    |
|   - verifying node conditions  |  0.002157 |           0.002095 |
| ...                            |           |                    |
|   - verifying api server ...   |  0.060020 |           0.092590 |
|   - verifying system pods ...  |  1.549097 |           1.339155 |
| * Enabled addons:              |  0.362699 |           0.727858 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.072873 |           0.084172 |
| configured to use "minikube"   |           |                    |
|                                |  0.002694 |           0.005145 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.084724851 26.21288166 25.282186997]
Average time for minikube: 26.19326450266667

Times for Minikube (PR 7793): [27.343876171 25.786497053999998 25.37853571]
Average time for Minikube (PR 7793): 26.169636311666665

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7793) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian            |  0.070216 |           0.072151 |
|                                   9.11 |           |                    |
| * Using the docker driver              |  0.002347 |           0.002254 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.054510 |           0.055502 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.406008 |           7.132159 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000154 |           0.000276 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.169012 |          17.339089 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.002036 |           0.001338 |
| Components:                            |           |                    |
|   - verifying node conditions          |  0.001591 |           0.003000 |
| ...                                    |           |                    |
|   - verifying api server ...           |  0.090229 |           0.076809 |
|   - verifying system pods ...          |  1.326277 |           1.414223 |
| * Enabled addons:                      |  0.002608 |           0.002446 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.064723 |           0.066444 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003553 |           0.003945 |
+----------------------------------------+-----------+--------------------+

Copy link

@priyawadhwa priyawadhwa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, a couple small comments

pkg/minikube/cruntime/containerd _test.go Outdated Show resolved Hide resolved
pkg/minikube/cruntime/containerd.go Show resolved Hide resolved
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [65.1175123 66.703359101 64.962278437]
Average time for minikube: 65.59438327933333

Times for Minikube (PR 7793): [61.107381781 65.221457668 63.370848020000004]
Average time for Minikube (PR 7793): 63.23322915633333

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7793) |
+--------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian    |  0.056625 |           0.054504 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.018960 |           0.018637 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.008054 |           0.004721 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.060120 |          40.560338 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 22.672359 |          20.953495 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.368235 |                    |
| components...                  |           |                    |
| * Enabled addons:              |  0.320140 |           0.324581 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.085242 |           0.067970 |
| configured to use "minikube"   |           |                    |
|                                |  0.004648 |           1.158078 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [25.469914597999995 26.206790298999998 26.001329295]
Average time for minikube: 25.892678063999995

Times for Minikube (PR 7793): [25.285381557000004 24.947371845000003 24.594557573]
Average time for Minikube (PR 7793): 24.942436991666664

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7793) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian            |  0.072031 |           0.071338 |
|                                   9.11 |           |                    |
| * Using the docker driver              |  0.002280 |           0.002195 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.054094 |           0.054067 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.428213 |           7.378511 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000161 |           0.000189 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.060052 |          16.421976 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.108853 |                    |
| components...                          |           |                    |
| * Enabled addons:                      |  0.093579 |           0.092027 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.069064 |           0.063514 |
| configured to use "minikube"           |           |                    |
|                                        |  0.004352 |           0.734443 |
+----------------------------------------+-----------+--------------------+

@priyawadhwa priyawadhwa merged commit 311547a into kubernetes:master Apr 20, 2020
@medyagh
Copy link
Member Author

medyagh commented Apr 20, 2020

@medyagh medyagh changed the title add preload feature for containerd add preload feature for containerd. faster containerd start. Apr 20, 2020
@medyagh medyagh changed the title add preload feature for containerd. faster containerd start. faster containerd start by preloading images Apr 20, 2020
@medyagh medyagh deleted the preload_containerd branch May 2, 2020 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants