Create a prototype of direct image build on containerd and cri-o

[medyagh]

What Happened?

we need to create a similar feature to docker-env that is only for docker-runtime.

[medyagh]

@afbjorklund knows a POC in his mind that could work, please share so someone could pick it up

[ComradeProgrammer]

/assign

[afbjorklund]

For accessing the podman.sock (for cri-o) and nerdctl.sock (for containerd), we need to use SSH.

• Deprecate the old tcp port for docker-env #9229

• docker/cli@6f61cf0 available since Docker Engine 18.09

So the first step, would be to make sure that using SSH for docker.sock works OK and is documented:

minikube ssh-host --append-known

• Add ssh-host command for getting the ssh host keys #9630

minikube docker-env --ssh-host --ssh-add

• Optionally use ssh for docker-env instead of tcp #9548

https://docs.docker.com/engine/security/protect-access/#use-ssh-to-protect-the-docker-daemon-socket

Then all the other runtimes need to do, is to provide a docker in PATH and docker.sock in DOCKER_HOST

[afbjorklund]

Typical output of docker-env tcp, 3 variables:

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:49156"
export DOCKER_CERT_PATH="/home/anders/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)

Typical output of docker-env ssh, 1 variable:

export DOCKER_HOST="ssh://[email protected]:49157"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env --ssh-host)

---

docker@minikube:~$ DOCKER_HOST=unix:///var/run/docker.sock docker version
Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:12 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[afbjorklund]

The nerdctl.sock is available from here: https://github.com/afbjorklund/nerdctld

docker@minikube:~$ DOCKER_HOST=unix:///var/run/nerdctl.sock docker version
Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.40
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 nerdctl:
  Version:          1.0.0
 buildctl:
  Version:          0.10.6
  GitCommit:        0c9b5aeb269c740650786ba77d882b0259415ec7
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
 Engine:
  Version:          1.0.0
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.19.2
  Git commit:       c00780a1f5b905b09812722459c54936c9e070e6
  Built:            
  OS/Arch:          linux/amd64
  Experimental:     true

Note that the unit CONTAINERD_NAMESPACE need to be changed to use "k8s.io"

The podman.sock should already be included with the existing images, for v3.4.2

docker@minikube:~$ DOCKER_HOST=unix:///run/podman/podman.sock docker version
Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.40
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: linux/amd64/ubuntu-20.04
 Podman Engine:
  Version:          3.4.2
  APIVersion:       3.4.2
  Arch:             amd64
  BuildTime:        1970-01-01T00:00:00Z
  Experimental:     false
  GitCommit:        
  GoVersion:        go1.15.2
  KernelVersion:    5.4.0-131-generic
  MinAPIVersion:    3.1.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.2, commit: 
  Package:          conmon: /usr/libexec/podman/conmon
 OCI Runtime (crun):
  Version:          crun version UNKNOWN
commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  Package:          crun: /usr/bin/crun
 Engine:
  Version:          3.4.2
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.15.2
  Git commit:       
  Built:            Thu Jan  1 00:00:00 1970
  OS/Arch:          linux/amd64
  Experimental:     false

[afbjorklund]

The default option in minikube should be to use the minikube image command instead of the legacy docker client...

https://minikube.sigs.k8s.io/docs/commands/image/

The "docker-env" command is mostly for backwards compatibility, and the "podman-env" should be fully deprecated.

It is also possible to do everything yourself with the minikube ssh command and minikube mount, but less usable.

---

The main use for "docker-env", is when using minikube as a replacement for Docker Toolbox and Docker Machine...

That is, as an OS alternative to Docker Desktop. Not when running a Kubernetes cluster, which doesn't need Docker.

docker-machine start
docker-machine env

minikube start --no-kubernetes
eval $(minikube docker-env)

So the docker commands in this issue, are only for minikube. Mostly filling in the gaps in the CRI specification ?

If you need to run containerd in a VM, there are other projects.

[afbjorklund]

As per the docker instructions, one should make sure to configure $DOCKER_HOST on the remote side

• https://docs.docker.com/engine/security/protect-access/#use-ssh-to-protect-the-docker-daemon-socket

• https://docs.docker.com/engine/security/rootless/#expose-docker-api-socket-through-ssh

So that when one connects via ssh, the environment variable points to the correct unix socket location.

Using the path parameter in the URL is not supported:

ssh host connection is not valid: extra path after the host: "/var/run/docker.sock"

A workaround is to create a symlink from the default location (/var/run/docker.sock), but more of a hack...

The same goes for setting up a tunneled local unix socket, just to avoid adding the identity and host keys.

---

So the best is to use a global configuration in /etc, to properly set the socket location for the selected runtime ?

k8s runtime	cri	crictl socket	client	docker socket
docker	cri-dockerd	/var/run/cri-dockerd.sock	docker	/var/run/docker.sock
cri-o	crio	/run/crio/crio.sock	podman	/run/podman/podman.sock
containerd	containerd*	/run/containerd/containerd.sock	nerdctl	/var/run/nerdctl.sock

Normally the variable contains the URL (with a unix:// scheme). In systemd (only), /var/run is a symlink to /run.

* requires the "cri" containerd plugin to be enabled

The node also needs to have a "docker" program installed.

The program doesn't have to do anything but be able to run the hardcoded docker system dial-stdio command...

If not having the real docker client installed on node, it is possible to use a small shell script called socat or similar.

Note that podman and nerdctl do not talk to their socket.

[afbjorklund]

See this issue, for the "nerdctl" installation:

• Add nerdctl command for users outside kubernetes #13252

It only needs the client, not the "nerdctl-full" install.

For building images, a "buildkit" installation is needed.

---

This is the full "containerd" runtime installation:

sbin/runc

bin/ctr
bin/containerd
bin/containerd-shim
bin/containerd-shim-runc-v1
bin/containerd-shim-runc-v2
lib/systemd/system/containerd.service

bin/buildctl
bin/buildkitd
lib/systemd/system/buildkit.service
lib/systemd/system/buildkit.socket

bin/nerdctl
bin/nerdctld
lib/systemd/system/nerdctl.service
lib/systemd/system/nerdctl.socket

The CRI plugin is included and enabled by default:

https://kubernetes.io/blog/2018/05/24/kubernetes-containerd-integration-goes-ga/

(it is disabled in the docker packaging, but not in the upstream default configuration)

The buildkitd and nerdctld are socket-activated, so only started when actually used.

[afbjorklund]

This is the full "cri-o" runtime installation:

bin/crun

bin/conmon

bin/crio
bin/crio-status
bin/pinns
lib/systemd/system/crio.service
lib/systemd/system/crio-wipe.service

bin/podman
lib/systemd/system/podman.service
lib/systemd/system/podman.socket

The "podman" binary is both client and server.

[afbjorklund]

Note: for Docker 23.0.x, you need to use DOCKER_BUILDKIT=0 when using nerdctl.sock.

It is somewhat ironic, since it will always use BuildKit - but it is not managed by Docker.