nixos/kubernetes: replace docker shim with CRI-O

[saschagrunert]

Motivation for this change

This replaces the default Kubernetes docker-shim with the more lightweight cri-o module.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[zowoq]

I'd suggest adding a nixos test for cri-o so it can be tested without kubernetes.

[saschagrunert]

I'd suggest adding a nixos test for cri-o so it can be tested without kubernetes.

Alright I can do that before continuing here. 👍

[saschagrunert]

Okay, so the bridge plugin (default) is working now, I'm just wondering what we wanna do with flannel. 🤔

[saschagrunert]

Looks like that we would have to change the CRI-O network_dir config to point to the new flannel config if flannel should be used. @zowoq do you have any idea how to do that properly? I can add the networkDir option to CRI-O in a different PR.

[saschagrunert]

This is ready for testing and review. PTAL @NixOS/podman @johanot @srhb

@GrahamcOfBorg test kubernetes.dns
@GrahamcOfBorg test kubernetes.rbac

[vdemeester]

@saschagrunert two small question

[zowoq]

Looks like there is some docker specific things in the kubernetes tests that probably need to be updated.

[srhb]

Hey sascha! This sounds very interesting, but I do have some questions, the number one being: "Why?" 😁 Don't get me wrong, there's no love lost between myself and Docker, but the current kubernetes ecosystem, while trying to move in a separate direction, does tend to give it special status, the blessed "default" so to say. Why should we move away from that?

That said, I'm definitely amenable to removing at least the special status of dockerized kubernetes in NixOS, making it up to the user to decide on their container runtime of choice. However, what about current users? I'm not very keen on making an, in the grand scale, "exotic" runtime the default. I'm sure we can work it out, but for instance our production cluster is currently docker, and we should try to figure out a migration strategy for users like us, unless we really decide to break everything for users, in which case we should at least have a good reason and document it thoroughly in the 21.03 release notes.

Looking forward to hearing your inputs. I really don't want to sound too pessimistic here; like I said I'm really not a fan of docker as such, but I think we should try and figure out some coherent goals here, so we can make a realistic game plan.

[saschagrunert]

Hey sascha! This sounds very interesting, but I do have some questions, the number one being: "Why?" Don't get me wrong, there's no love lost between myself and Docker, but the current kubernetes ecosystem, while trying to move in a separate direction, does tend to give it special status, the blessed "default" so to say. Why should we move away from that?

No worries I think the concerns are appropriate. The main reason for me personally was that I see docker and Kubernetes as two different things. Docker is the development tool which a much broader set of features than required to run Kubernetes. There is in-tree dockershim in Kubernetes, which is another set of glue code to make them work together.

The container runtime interface (CRI) has been developed to have exchangeable runtime implementations which follow a common standard. Right now we have two main runtimes available: containerd (originates historically from docker/moby) and CRI-O. Later on we could make them switch-able by configuration.

The dockershim will be deprecated and is going to be removed from Kubernetes if the corresponding KEP will be implemented:
kubernetes/enhancements#1985

That said, I'm definitely amenable to removing at least the special status of dockerized kubernetes in NixOS, making it up to the user to decide on their container runtime of choice. However, what about current users? I'm not very keen on making an, in the grand scale, "exotic" runtime the default. I'm sure we can work it out, but for instance our production cluster is currently docker, and we should try to figure out a migration strategy for users like us, unless we really decide to break everything for users, in which case we should at least have a good reason and document it thoroughly in the 21.03 release notes.

Yeah I thought about making it configurable, which definitely would increase the complexity of the derivation, but should work in general.

[Mic92]

Dockershim beeing deprecated in kubernetes sounds like a strong argument to me.

[vdemeester]

I tend to agree with @saschagrunert and @Mic92 but indeed @srhb, having a migration path (so having a way for the current user to not get broken) is most likely required 👼. Making it configurable with a default to docker to start and go on a deprecation path seems the right thing to do, even if it temporarly increase the complexity of the derivation.

Small note, I wouldn't call cri-o exotic — it is widely used, on all OpenShift 4 cluster running. Similar to containerd which is also quite well battle tested (I think google cloud uses it for GKE by default now).

[srhb]

Small note, I wouldn't call cri-o exotic — it is widely used, on all OpenShift 4 cluster running. Similar to containerd which is also quite well battle tested (I think google cloud uses it for GKE by default now).

Sorry, I was being a bit contrary with "exotic" for the purpose of illustrating my concerns. 😁

I largely agree with everything else, including especially accepting the increased complexity in order to remain compatible.

[saschagrunert]

Sounds good, I'll rework this PR to make docker the default but be able to exchange the runtime

[aaronjanse]

EDIT: Ah, it was fixed by removing all my kubernetes stuff in /etc/kubernetes

Unsure if the problem is on my end, but after applying this PR locally, coredns fails to install, blocking user-created pods from starting

Dec 27 01:50:05 ajanse-xps kubelet[2557]: E1227 01:50:05.901819    2557 remote_image.go:113] PullImage "coredns/coredns:1.6.4" from image service failed: rpc error: code = Unknown desc = Requesting bear token: invalid status code from registry 405 (Method Not Allowed)
Dec 27 01:50:05 ajanse-xps kubelet[2557]: E1227 01:50:05.901862    2557 kuberuntime_image.go:51] Pull image "coredns/coredns:1.6.4" failed: rpc error: code = Unknown desc = Requesting bear token: invalid status code from registry 405 (Method Not Allowed)
Dec 27 01:50:05 ajanse-xps kubelet[2557]: E1227 01:50:05.902047    2557 kuberuntime_manager.go:804] container &Container{Name:coredns,Image:coredns/coredns:1.6.4,Command:[],Args:[-conf /etc/coredns/Corefile],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:dns,HostPort:0,ContainerPort:10053,Protocol:UDP,HostIP:,},ContainerPort{Name:dns-tcp,HostPort:0,ContainerPort:10053,Protocol:TCP,HostIP:,},ContainerPort{Name:metrics,HostPort:0,ContainerPort:10055,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{memory: {{178257920 0} {<nil>} 170Mi BinarySI},},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{73400320 0} {<nil>} 70Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:config-volume,ReadOnly:true,MountPath:/etc/coredns,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:coredns-token-65vf7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:{0 10054 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:5,},ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[all],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod coredns-77c8759db9-kvqxk_kube-system(09a0ae92-cdf0-4722-9f37-fd085c628faa): ErrImagePull: rpc error: code = Unknown desc = Requesting bear token: invalid status code from registry 405 (Method Not Allowed)
Dec 27 01:50:05 ajanse-xps kubelet[2557]: E1227 01:50:05.902073    2557 pod_workers.go:191] Error syncing pod 09a0ae92-cdf0-4722-9f37-fd085c628faa ("coredns-77c8759db9-kvqxk_kube-system(09a0ae92-cdf0-4722-9f37-fd085c628faa)"), skipping: failed to "StartContainer" for "coredns" with ErrImagePull: "rpc error: code = Unknown desc = Requesting bear token: invalid status code from registry 405 (Method Not Allowed)"
Dec 27 01:50:05 ajanse-xps crio[2479]: time="2020-12-27 01:50:05.902065793-08:00" level=info msg="Pulling image: coredns/coredns:1.6.4" id=f93276a1-3740-4942-a844-0621c1afcfc1 name=/runtime.v1alpha2.ImageService/PullImage

[saschagrunert]

Let's close this PR for now, I know that there are ongoing efforts in replacing docker with a configurable variant (containerd and CRI-O).