Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a prototype of direct image build on containerd and cri-o #15389

Closed
medyagh opened this issue Nov 21, 2022 · 10 comments · Fixed by #15452
Closed

Create a prototype of direct image build on containerd and cri-o #15389

medyagh opened this issue Nov 21, 2022 · 10 comments · Fixed by #15452
Assignees
Labels
co/docker-env docker-env issues help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@medyagh
Copy link
Member

medyagh commented Nov 21, 2022

What Happened?

we need to create a similar feature to docker-env that is only for docker-runtime.

@medyagh
Copy link
Member Author

medyagh commented Nov 21, 2022

@afbjorklund knows a POC in his mind that could work, please share so someone could pick it up

@medyagh medyagh added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Nov 21, 2022
@ComradeProgrammer
Copy link
Member

/assign

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 21, 2022

For accessing the podman.sock (for cri-o) and nerdctl.sock (for containerd), we need to use SSH.

So the first step, would be to make sure that using SSH for docker.sock works OK and is documented:

minikube ssh-host --append-known

minikube docker-env --ssh-host --ssh-add

https://docs.docker.com/engine/security/protect-access/#use-ssh-to-protect-the-docker-daemon-socket

Then all the other runtimes need to do, is to provide a docker in PATH and docker.sock in DOCKER_HOST

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 21, 2022

Typical output of docker-env tcp, 3 variables:

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:49156"
export DOCKER_CERT_PATH="/home/anders/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)

Typical output of docker-env ssh, 1 variable:

export DOCKER_HOST="ssh://[email protected]:49157"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env --ssh-host)

docker@minikube:~$ DOCKER_HOST=unix:///var/run/docker.sock docker version
Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:12 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 22, 2022

The nerdctl.sock is available from here: https://github.com/afbjorklund/nerdctld

docker@minikube:~$ DOCKER_HOST=unix:///var/run/nerdctl.sock docker version
Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.40
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 nerdctl:
  Version:          1.0.0
 buildctl:
  Version:          0.10.6
  GitCommit:        0c9b5aeb269c740650786ba77d882b0259415ec7
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
 Engine:
  Version:          1.0.0
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.19.2
  Git commit:       c00780a1f5b905b09812722459c54936c9e070e6
  Built:            
  OS/Arch:          linux/amd64
  Experimental:     true

Note that the unit CONTAINERD_NAMESPACE need to be changed to use "k8s.io"

The podman.sock should already be included with the existing images, for v3.4.2

docker@minikube:~$ DOCKER_HOST=unix:///run/podman/podman.sock docker version
Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.40
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: linux/amd64/ubuntu-20.04
 Podman Engine:
  Version:          3.4.2
  APIVersion:       3.4.2
  Arch:             amd64
  BuildTime:        1970-01-01T00:00:00Z
  Experimental:     false
  GitCommit:        
  GoVersion:        go1.15.2
  KernelVersion:    5.4.0-131-generic
  MinAPIVersion:    3.1.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.2, commit: 
  Package:          conmon: /usr/libexec/podman/conmon
 OCI Runtime (crun):
  Version:          crun version UNKNOWN
commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  Package:          crun: /usr/bin/crun
 Engine:
  Version:          3.4.2
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.15.2
  Git commit:       
  Built:            Thu Jan  1 00:00:00 1970
  OS/Arch:          linux/amd64
  Experimental:     false

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 24, 2022

The default option in minikube should be to use the minikube image command instead of the legacy docker client...

https://minikube.sigs.k8s.io/docs/commands/image/

The "docker-env" command is mostly for backwards compatibility, and the "podman-env" should be fully deprecated.

It is also possible to do everything yourself with the minikube ssh command and minikube mount, but less usable.


The main use for "docker-env", is when using minikube as a replacement for Docker Toolbox and Docker Machine...

That is, as an OS alternative to Docker Desktop. Not when running a Kubernetes cluster, which doesn't need Docker.

docker-machine start
docker-machine env
minikube start --no-kubernetes
eval $(minikube docker-env)

So the docker commands in this issue, are only for minikube. Mostly filling in the gaps in the CRI specification ?

If you need to run containerd in a VM, there are other projects.

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 3, 2022

As per the docker instructions, one should make sure to configure $DOCKER_HOST on the remote side

So that when one connects via ssh, the environment variable points to the correct unix socket location.

Using the path parameter in the URL is not supported:

ssh host connection is not valid: extra path after the host: "/var/run/docker.sock"

A workaround is to create a symlink from the default location (/var/run/docker.sock), but more of a hack...

The same goes for setting up a tunneled local unix socket, just to avoid adding the identity and host keys.


So the best is to use a global configuration in /etc, to properly set the socket location for the selected runtime ?

k8s runtime cri crictl socket client docker socket
docker cri-dockerd /var/run/cri-dockerd.sock docker /var/run/docker.sock
cri-o crio /run/crio/crio.sock podman /run/podman/podman.sock
containerd containerd* /run/containerd/containerd.sock nerdctl /var/run/nerdctl.sock

Normally the variable contains the URL (with a unix:// scheme). In systemd (only), /var/run is a symlink to /run.

* requires the "cri" containerd plugin to be enabled

The node also needs to have a "docker" program installed.

The program doesn't have to do anything but be able to run the hardcoded docker system dial-stdio command...

If not having the real docker client installed on node, it is possible to use a small shell script called socat or similar.

Note that podman and nerdctl do not talk to their socket.

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 3, 2022

See this issue, for the "nerdctl" installation:

It only needs the client, not the "nerdctl-full" install.

For building images, a "buildkit" installation is needed.


This is the full "containerd" runtime installation:

sbin/runc

bin/ctr
bin/containerd
bin/containerd-shim
bin/containerd-shim-runc-v1
bin/containerd-shim-runc-v2
lib/systemd/system/containerd.service

bin/buildctl
bin/buildkitd
lib/systemd/system/buildkit.service
lib/systemd/system/buildkit.socket

bin/nerdctl
bin/nerdctld
lib/systemd/system/nerdctl.service
lib/systemd/system/nerdctl.socket

The CRI plugin is included and enabled by default:

https://kubernetes.io/blog/2018/05/24/kubernetes-containerd-integration-goes-ga/

(it is disabled in the docker packaging, but not in the upstream default configuration)

The buildkitd and nerdctld are socket-activated, so only started when actually used.

@afbjorklund afbjorklund changed the title Create a prototype of direct image build on containerd Create a prototype of direct image build on containerd and cri-o Dec 3, 2022
@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 5, 2022

This is the full "cri-o" runtime installation:

bin/crun

bin/conmon

bin/crio
bin/crio-status
bin/pinns
lib/systemd/system/crio.service
lib/systemd/system/crio-wipe.service

bin/podman
lib/systemd/system/podman.service
lib/systemd/system/podman.socket

The "podman" binary is both client and server.

@afbjorklund
Copy link
Collaborator

afbjorklund commented Apr 11, 2023

Note: for Docker 23.0.x, you need to use DOCKER_BUILDKIT=0 when using nerdctl.sock.

It is somewhat ironic, since it will always use BuildKit - but it is not managed by Docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/docker-env docker-env issues help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants