Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate cri-o container runtime preload tarball #8581

Merged
merged 6 commits into from
Jul 20, 2020

Conversation

afbjorklund
Copy link
Collaborator

@afbjorklund afbjorklund commented Jun 27, 2020

Added cri-o, to the list of preloaded tarballs:

A preloaded tarball for k8s version v1.18.3 - runtime "docker" already exists, skipping generation.
A preloaded tarball for k8s version v1.18.3 - runtime "containerd" already exists, skipping generation.
A preloaded tarball for k8s version v1.18.3 - runtime "cri-o" doesn't exist, generating now...
Image is up to date for k8s.gcr.io/kube-proxy@sha256:6a093c22e305039b7bd6c3f8eab8f202ad8238066ed210857b25524443aa8aff
Image is up to date for k8s.gcr.io/kube-scheduler@sha256:5381cd9680bf5fb16a5c8ac60141eaab242c1c4960f1c32a21807efcca3e765b
Image is up to date for k8s.gcr.io/kube-controller-manager@sha256:168e8f9276e5f947d68f93a9463243f6a525227d15f55fb40b427c8d6caa55c2
Image is up to date for k8s.gcr.io/kube-apiserver@sha256:b5bf4650d7f084d0c9f89ecae50c00ebf5b44edcdd487636d8c7fd400d27cd05
Image is up to date for k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800
Image is up to date for k8s.gcr.io/etcd@sha256:4198ba6f82f642dfd18ecf840ee37afb9df4b596f06eef20e44d0aec4ea27216
Image is up to date for k8s.gcr.io/pause@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
Image is up to date for gcr.io/k8s-minikube/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a
Image is up to date for docker.io/kubernetesui/dashboard@sha256:19207cca570f61bce2294595a767b51b4d7e1f223a003c7972c350cf8689f49e
Image is up to date for docker.io/kubernetesui/metrics-scraper@sha256:555981a24f184420f3be0c79d4efb6c948a85cfce84034f85a563f4151a81cbf
Image is up to date for docker.io/kindest/kindnetd@sha256:46e34ccb3e08557767b7c80e957741d9f2590968ff32646875632d40cf62adad

For #6934

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 27, 2020
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2020
@codecov-commenter
Copy link

codecov-commenter commented Jun 27, 2020

Codecov Report

Merging #8581 into master will decrease coverage by 0.08%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8581      +/-   ##
==========================================
- Coverage   32.14%   32.05%   -0.09%     
==========================================
  Files         161      161              
  Lines       10494    10523      +29     
==========================================
  Hits         3373     3373              
- Misses       6717     6746      +29     
  Partials      404      404              
Impacted Files Coverage Δ
pkg/minikube/cruntime/crio.go 46.22% <0.00%> (-7.04%) ⬇️
pkg/addons/addons.go 40.95% <0.00%> (-3.16%) ⬇️

Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you plz test it manually ? You can generate preload tar and then put it in the cache folder and if it works successfully we will merge it and let the cron job create the tars

@afbjorklund
Copy link
Collaborator Author

It seems to auto-upload, so I guess I need to comment that out... And need to determine if that “cache” folder is required by the “storage”

So maybe need the preload part as well, to see whether it actually works ?

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jun 27, 2020

I included everything for now, until I actually know what is in those databases...

./lib/containers/sigstore/
./lib/containers/storage/storage.lock
./lib/containers/storage/overlay-layers/*
./lib/containers/mounts/
./lib/containers/storage/overlay/*
./lib/containers/storage/overlay-images/*
./lib/containers/cache/blob-info-cache-v1.boltdb

Note that the "sigstore" and "mounts" are empty, so it's just about "cache".

Seems to have worked otherwise. Tested with just the default k8s version.

592M	out/preloaded-images-k8s-v4-v1.18.3-cri-o-overlay2-amd64.tar.lz4

Note: that this Step 1 was only about creating the tarball, not about using it

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jun 27, 2020

The name "overlay2" seems to be hard-coded. It was supposed to be "overlay"...
It's the same thing, anyway. Docker just renamed theirs, since they had overlay1

// TarballName returns name of the tarball
func TarballName(k8sVersion, containerRuntime string) string {
        return fmt.Sprintf("preloaded-images-k8s-%s-%s-%s-overlay2-%s.tar.lz4", PreloadVersion, k8sVersion, containerRuntime, runtime.GOARCH)
}

But this name is actually implemented by "download", so it will be in Step 2: loading
It can also stay as overlay2 in the file name, I'm not sure if anyone actually cares ?

driver=""
container storage driver (default: "overlay")
Default Copy On Write (COW) container storage driver. Valid drivers are "overlay", "vfs", "devmapper", "aufs", "btrfs", and "zfs". Some drivers (for example, "zfs", "btrfs", and "aufs") may not work if your kernel lacks support for the filesystem.

Fixed the name from the start. The actual API is a bit stupid (uses strings), so still hard-coded.

       if containerRuntime == "cri-o" {
               storageDriver = "overlay"
       } else {
               storageDriver = "overlay2"
       }

@afbjorklund afbjorklund changed the title Generate cri-o container runtime preload tarball WIP: Generate cri-o container runtime preload tarball Jun 28, 2020
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 28, 2020
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 28, 2020
@afbjorklund afbjorklund changed the title WIP: Generate cri-o container runtime preload tarball Generate cri-o container runtime preload tarball Jun 28, 2020
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 28, 2020
@afbjorklund
Copy link
Collaborator Author

New name:

592M	out/preloaded-images-k8s-v4-v1.18.3-cri-o-overlay-amd64.tar.lz4

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jun 28, 2020

@medyagh : trying to use preload with podman currently fails with the usual problem of overloading /var:

I0628 14:36:38.585784  176417 cli_runner.go:109] Run: sudo -n podman logs --timestamps crio
I0628 14:36:38.657245  176417 errors.go:60] Postmortem logs ("sudo -n podman logs --timestamps crio"): -- stdout --
2020-06-28T14:36:24.146928527+02:00 INFO: ensuring we can execute /bin/mount even with userns-remap
2020-06-28T14:36:24.164124815+02:00 INFO: remounting /sys read-only
2020-06-28T14:36:24.165968352+02:00 INFO: making mounts shared
2020-06-28T14:36:24.167145397+02:00 INFO: fix cgroup mounts for all subsystems
2020-06-28T14:36:24.169329664+02:00 INFO: clearing and regenerating /etc/machine-id
2020-06-28T14:36:24.174558308+02:00 Initializing machine ID from random generator.
2020-06-28T14:36:24.197085721+02:00 INFO: faking /sys/class/dmi/id/product_name to be "kind"
2020-06-28T14:36:24.199985019+02:00 INFO: faking /sys/class/dmi/id/product_uuid to be random
2020-06-28T14:36:24.201451595+02:00 INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
2020-06-28T14:36:24.215181859+02:00 INFO: setting iptables to detected mode: legacy
2020-06-28T14:36:24.215822257+02:00 update-alternatives: error: no alternatives for iptables

So we should fix that one (#8056 #8100 #8179) first, before enabling this feature. But we need these tarballs, first.

EDIT: This is only a problem for the KIC drivers, we can still use preloaded tarballs with the VM drivers

Pretty much the same as containerd, minus Restart

(cherry picked from commit d9fb2e0)
hack/preload-images/preload_images.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 30, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@afbjorklund
Copy link
Collaborator Author

Tested locally, seems to work ok:

962M	out/preloaded-images-k8s-v4-v1.18.3-containerd-overlay2-amd64.tar.lz4
592M	out/preloaded-images-k8s-v4-v1.18.3-cri-o-overlay-amd64.tar.lz4
527M	out/preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4

@afbjorklund afbjorklund requested a review from medyagh June 30, 2020 16:58
@TravisBuddy
Copy link

Travis tests have failed

Hey @afbjorklund,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.13.9.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
/home/travis/gopath/bin/go-bindata
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.13.9.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.12.0-beta.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.11.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="ad6e504fe38b7b4e4b7673242a8038dd66b5c0e1"" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.26.0'
golangci/golangci-lint info found version: 1.26.0 for v1.26.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
pkg/minikube/cruntime/cri.go:269: File is not `goimports`-ed (goimports)
        if !strings.Contains(imgName, ".io/") {
                return "docker.io/" + imgName
        } // else it already has repo name dont add anything
        return imgName
Makefile:390: recipe for target 'lint-ci' failed
make[1]: *** [lint-ci] Error 1
= go mod ================================================================
ok
= boilerplate ===========================================================
ok
Makefile:274: recipe for target 'test' failed
make: *** [test] Error 4
TravisBuddy Request Identifier: a9c15360-baf3-11ea-bd85-f15c2d376ac7

Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you

@medyagh medyagh merged commit 79088e6 into kubernetes:master Jul 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants