Unable to start minikube using Docker driver and CRI-O runtime on Fedora32 (cgroups v1) #9625

vrubiolo · 2020-11-06T22:42:44Z

Steps to reproduce the issue:

As per our discussion with @tstromberg on Slack, I am opening this issue.

minikube start --alsologtostderr --driver=docker --container-runtime=crio -v=1
Minikube does not start

This works with minikube start --driver=virtualbox --container-runtime=crio

Full output of failed command:
minikube-crio-log.txt

Output of minikube ssh "sudo journalctl -u crio"

-- Logs begin at Fri 2020-11-06 22:32:58 UTC, end at Fri 2020-11-06 22:39:21 UTC. --
Nov 06 22:33:00 minikube systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Nov 06 22:33:00 minikube systemd[1]: Started Container Runtime Interface for OCI (CRI-O).

Output of minikube ssh "sudo journalctl -u kubelet --no-pager" (12MiB large)
minikube-crio-kubelet-log.txt.gz

This is Fedora 32, configured w/ cgroups v1:

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.8.16-200.fc32.x86_64 root=/dev/mapper/fedora_localhost--live-root ro resume=/dev/mapper/fedora_localhost--live-swap rd.lvm.lv=fedora_localhost-live/root rd.lvm.lv=fedora_localhost-live/swap rhgb quiet systemd.unified_cgroup_hierarchy=0

$ minikube version
minikube version: v1.14.2
commit: 2c82918e2347188e21c4e44c8056fc80408bce10

The text was updated successfully, but these errors were encountered:

afbjorklund · 2020-11-07T08:07:13Z

Looks like the same crio bug as before:

Nov 06 22:33:15 minikube kubelet[796]: F1106 22:33:15.404404 796 kubelet.go:1296] Failed to start ContainerManager failed to initialize top level QOS containers: root container [kubepods] doesn't exist

Reported upstream as cri-o/cri-o#4035

* Preparing Kubernetes v1.19.2 on CRI-O 1.18.3 ...

Nov 06 22:33:07 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.

Nov 06 22:33:15 minikube kubelet[796]: E1106 22:33:15.404371     796 cgroup_manager_linux.go:674] cgroup update failed failed to set supported cgroup subsystems for cgroup [kubepods]: failed to set config for supported subsystems : failed to write "16423137280" to "/sys/fs/cgroup/memory/kubepods.slice/memory.limit_in_bytes": open /sys/fs/cgroup/memory/kubepods.slice/memory.limit_in_bytes: no such file or directory
Nov 06 22:33:15 minikube kubelet[796]: F1106 22:33:15.404404     796 kubelet.go:1296] Failed to start ContainerManager failed to initialize top level QOS containers: root container [kubepods] doesn't exist
Nov 06 22:33:15 minikube kubelet[796]: goroutine 319 [running]:
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00012a001, 0xc001212a00, 0xa7, 0xf7)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:996 +0xb9
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x6cf6140, 0xc000000003, 0x0, 0x0, 0xc0006f7b90, 0x6b4aa40, 0xa, 0x510, 0x0)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:945 +0x191
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printf(0x6cf6140, 0xc000000003, 0x0, 0x0, 0x465fbf5, 0x23, 0xc000d8dcb0, 0x1, 0x1)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x17a
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1456
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initializeRuntimeDependentModules(0xc0007a0a80)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1296 +0x385
Nov 06 22:33:15 minikube kubelet[796]: sync.(*Once).doSlow(0xc0007a12b8, 0xc000ab9dd0)
Nov 06 22:33:15 minikube kubelet[796]:         /usr/local/go/src/sync/once.go:66 +0xec
Nov 06 22:33:15 minikube kubelet[796]: sync.(*Once).Do(...)
Nov 06 22:33:15 minikube kubelet[796]:         /usr/local/go/src/sync/once.go:57
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateRuntimeUp(0xc0007a0a80)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2119 +0x554
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000f4d0a0)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000f4d0a0, 0x4becee0, 0xc000a28120, 0x415e6455bdce601, 0xc00010e0c0)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000f4d0a0, 0x12a05f200, 0x0, 0xb5d98d0cba8fb601, 0xc00010e0c0)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
Nov 06 22:33:15 minikube kubelet[796]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000f4d0a0, 0x12a05f200, 0xc00010e0c0)
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
Nov 06 22:33:15 minikube kubelet[796]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run
Nov 06 22:33:15 minikube kubelet[796]:         /workspace/anago-v1.19.2-rc.0.12+19706d90d87784/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1343 +0x16a
Nov 06 22:33:15 minikube systemd[1]: �[0;1;39m�[0;1;31m�[0;1;39mkubelet.service: Main process exited, code=exited, status=255/EXCEPTION�[0m
Nov 06 22:33:15 minikube systemd[1]: �[0;1;38;5;185m�[0;1;39m�[0;1;38;5;185mkubelet.service: Failed with result 'exit-code'.�[0m

Nov 06 22:33:16 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Nov 06 22:33:16 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.

afbjorklund · 2020-11-07T08:26:01Z

There was a cri-o workaround previously: 61583d6

~~It was reverted recently, for some reason: 8b1def0~~

I think the ISO had the previous version, the KIC was updated ahead-of-time:

🔥  Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🎁  Preparing Kubernetes v1.19.2 on CRI-O 1.17.3 ...

This is tracked as #8861(follow up from #8854)

I don't think it is resolved yet ? Can try 1.18.4...

@sharifelgamal @medyagh

afbjorklund · 2020-11-07T08:27:48Z

A similar workaround for the docker driver, would be to change the yum configuration and downgrade to cri-o 1.17.3

It's also possible that a sudo systemctl restart crio would also fix it, but maybe the self-destruct feature kicked in first

afbjorklund · 2020-11-07T12:56:37Z

@vrubiolo

There is an updated image in PR #9629, if you like to build it and try it ?

Works OK on Ubuntu, at least. Haven't tried it myself on Fedora, though.

vrubiolo · 2020-11-07T13:31:38Z

@afbjorklund : sure, thanks for the suggestion, let me try that over the weekend and report back.

afbjorklund · 2020-11-07T16:34:01Z

Not sure why there would be any difference, since we're using the same ubuntu 20.04 distribution inside the docker container

but there would be different kernels involved, and I don't what caused the cri-o problem (occasionally?) in the first place anyway

afbjorklund · 2020-11-07T16:48:06Z

But I did change runc, since I suspect that it is involved in this - and it was recommended to do so by cri-o upstream...

https://bugzilla.redhat.com/show_bug.cgi?id=1508040#c15

The issue is caused by a race condition in the runc library.

Upgraded from 1.0.0-rc10 to 1.0.0-rc92

Note: We include cri-o-runc because Ubuntu and Debian include their own packaged version of runc. While this version should work with CRI-O, keeping the packaged versions of CRI-O and runc in sync ensures they work together.

vrubiolo · 2020-11-07T21:21:24Z

Hi @afbjorklund and thanks for being so reactive!

I am pleased to report that #9629 appears to fix the issue for me. I can now start minikube w/ CRI-O and the Docker driver:

(kicbase-crio)$ git log -1
commit c2d435945638631faa3f4d200f2895d4852125ea (HEAD -> kicbase-crio, origin/kicbase-crio)
Author: Anders F Björklund <[email protected]>
Date:   Sat Nov 7 12:52:43 2020 +0100

    Upgrade cri-o in kicbase to version 1.18.4
    
    Use the latest 1.18, since 1.18.4 is broken again <sigh>
    
    Use the recommended cri-o-runc package instead of runc

(kicbase-crio)$ make
[build went fine]
(kicbase-crio)$ make kic-base-image
[new Docker image generated]

(kicbase-crio)$ ./out/minikube start --driver=docker --base-image=local/kicbase:latest --container-runtime=cri-o --preload=false
😄  minikube v1.14.2 on Fedora 32
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🎁  Preparing Kubernetes v1.19.2 on CRI-O 1.18.4 ...
    > kubeadm.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 65 B / 65 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 41.01 MiB / 41.01 MiB [----------------] 100.00% 9.40 MiB p/s 4s
    > kubeadm: 37.30 MiB / 37.30 MiB [----------------] 100.00% 8.54 MiB p/s 4s
    > kubelet: 104.88 MiB / 104.88 MiB [-------------] 100.00% 12.26 MiB p/s 9s
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

(kicbase-crio)$ kubectl get pods
No resources found in default namespace.

(kicbase-crio)$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-f9fd979d6-h8xtz            1/1     Running   0          57s
kube-system   etcd-minikube                      0/1     Running   0          57s
kube-system   kindnet-5tc64                      1/1     Running   0          57s
kube-system   kube-apiserver-minikube            1/1     Running   0          57s
kube-system   kube-controller-manager-minikube   0/1     Running   0          57s
kube-system   kube-proxy-gt9hb                   1/1     Running   0          57s
kube-system   kube-scheduler-minikube            0/1     Running   0          57s
kube-system   storage-provisioner                1/1     Running   0          65s

./out/minikube start --driver=docker --base-image=local/kicbase:latest --container-runtime=cri-o --preload=false --alsologtostderr -v=1 logs
minikube-crio-fix.txt

vrubiolo · 2020-11-10T10:22:50Z

Thanks for merging #9629 in!

What is the process for closing an issue like the present one? Should I wait until the next minikube release or do we immediately close it?

medyagh · 2020-11-12T03:49:50Z

Thanks for merging #9629 in!

What is the process for closing an issue like the present one? Should I wait until the next minikube release or do we immediately close it?

thanks we can close this issue we are planning to release tomorrow, hope that be helpful

vrubiolo · 2020-11-12T08:40:07Z

Cool, no worries, will be happy to try the new release when it's out!

vrubiolo · 2020-11-16T09:01:26Z

I confirm the 1.15.0 release allows to start/use CRI-O w/ the Docker driver w/o any issues on Fedora32 now (via minikube start --container-runtime=crio).

Thanks for your help fixing this !

afbjorklund added co/docker-driver Issues related to kubernetes in container os/linux co/runtime/crio CRIO related issues labels Nov 7, 2020

afbjorklund added the kind/bug Categorizes issue or PR as related to a bug. label Nov 7, 2020

medyagh closed this as completed Nov 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to start minikube using Docker driver and CRI-O runtime on Fedora32 (cgroups v1) #9625

Unable to start minikube using Docker driver and CRI-O runtime on Fedora32 (cgroups v1) #9625

vrubiolo commented Nov 6, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020

afbjorklund commented Nov 7, 2020

vrubiolo commented Nov 7, 2020

afbjorklund commented Nov 7, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading

vrubiolo commented Nov 7, 2020

vrubiolo commented Nov 10, 2020

medyagh commented Nov 12, 2020

vrubiolo commented Nov 12, 2020

vrubiolo commented Nov 16, 2020

Unable to start minikube using Docker driver and CRI-O runtime on Fedora32 (cgroups v1) #9625

Unable to start minikube using Docker driver and CRI-O runtime on Fedora32 (cgroups v1) #9625

Comments

vrubiolo commented Nov 6, 2020 • edited Loading

afbjorklund commented Nov 7, 2020 • edited Loading

afbjorklund commented Nov 7, 2020 • edited Loading

afbjorklund commented Nov 7, 2020

afbjorklund commented Nov 7, 2020

vrubiolo commented Nov 7, 2020

afbjorklund commented Nov 7, 2020 • edited Loading

afbjorklund commented Nov 7, 2020 • edited Loading

vrubiolo commented Nov 7, 2020

vrubiolo commented Nov 10, 2020

medyagh commented Nov 12, 2020

vrubiolo commented Nov 12, 2020

vrubiolo commented Nov 16, 2020

vrubiolo commented Nov 6, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading

afbjorklund commented Nov 7, 2020 •

edited

Loading