❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ #807

manumartin · 2021-10-18T17:05:23Z

❗ TL;DR: Docker >=v20.10.5 is required for k3d v5.x ❗

What did you do

I tried to create a k3d cluster with k3d 5.x but it fails. Its working with 4.x

How was the cluster created?
- sudo k3d cluster create MYCLUSTER --trace --verbose
What did you do afterwards?
It seems like some kind of k3d/docker integration issue. I have reproduced the error message by using this docker-compose.yml:

test-container:
    image: rancher/k3d-tools:5.0.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    tmpfs:
      - /var/run

It seems you can't mount the docker sock volume inside the /var/run tmpfs mount and k3d is failing when trying to do the same with the k3d-tools container.

What did you expect to happen

I expected the cluster to start normally

Screenshots or terminal output

❯ sudo k3d cluster create MYCLUSTER --trace --verbose

...

INFO[0000] Starting new tools node...                   
TRAC[0000] Creating node from spec
&{Name:k3d-MYCLUSTER-tools Role:noRole Image:docker.io/rancher/k3d-tools:5.0.0 Volumes:[k3d-MYCLUSTER-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: RuntimeLabels:map[app:k3d k3d.cluster:MYCLUSTER k3d.version:v5.0.0] K3sNodeLabels:map[] Networks:[k3d-MYCLUSTER] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-MYCLUSTER-tools Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[noop] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-tools:5.0.0 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:MYCLUSTER k3d.role:noRole k3d.version:v5.0.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-MYCLUSTER-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00020630f} NetworkingConfig:{EndpointsConfig:map[k3d-MYCLUSTER:0xc00024e000]}} 
DEBU[0000] Created container k3d-MYCLUSTER-tools (ID: 6aed150173f55ae288a9334302ccb6684dda8e9bdc1bc77af3454e4ab8743c64) 
DEBU[0000] Node k3d-MYCLUSTER-tools Start Time: 2021-10-18 17:15:55.925990393 +0200 CEST m=+0.122850186 
TRAC[0000] Starting node 'k3d-MYCLUSTER-tools'               
INFO[0000] Starting Node 'k3d-MYCLUSTER-tools'               
ERRO[0000] Failed to run tools container for cluster 'MYCLUSTER' 

...

INFO[0001] Starting existing tools node k3d-devo-tools... 
INFO[0001] Starting Node 'k3d-devo-tools'               
ERRO[0001] failed to gather environment information used for cluster creation: error starting existing tools node k3d-devo-tools: docker failed to start container for node 'k3d-devo-tools': Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/var/run/docker.sock" to rootfs at "/var/run/docker.sock" caused: stat /var/run/docker.sock: no such file or directory: unknown 
ERRO[0001] Failed to create cluster >>> Rolling Back

Which OS & Architecture

Linux x86

Which version of `k3d`

k3d version v5.0.1
k3s version v1.21.5-k3s1 (default)`

Which version of docker

Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:32 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

The text was updated successfully, but these errors were encountered:

manumartin · 2021-10-20T09:00:44Z

It doesn't happen anymore after upgrading from Docker 2.10.2 and containerd 1.4.3 to Docker 2.10.9 and containerd 1.4.11

franpog859 · 2021-10-24T20:15:54Z

Hey! I'm facing similar issue on Mac:

$ k3d cluster create mycluster

INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster'
INFO[0000] Created volume 'k3d-mycluster-images'
INFO[0000] Starting new tools node...
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0002] Pulling image 'docker.io/rancher/k3d-tools:5.0.1'
INFO[0003] Pulling image 'docker.io/rancher/k3s:latest'
INFO[0006] Starting Node 'k3d-mycluster-tools'
ERRO[0007] Failed to run tools container for cluster 'mycluster'
INFO[0031] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0033] Pulling image 'docker.io/rancher/k3d-proxy:5.0.1'
INFO[0041] Using the k3d-tools node to gather environment information
INFO[0041] Starting existing tools node k3d-mycluster-tools...
INFO[0041] Starting Node 'k3d-mycluster-tools'
ERRO[0041] failed to gather environment information used for cluster creation: error starting existing tools node k3d-mycluster-tools: docker failed to start container for node 'k3d-mycluster-tools': Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/run/host-services/docker.proxy.sock\\\" to rootfs \\\"/var/lib/docker/overlay2/7fd7682bb903b7768c61cc827cb4638f13568661f1c5b3ab007b9e289ebca6eb/merged\\\" at \\\"/var/run/docker.sock\\\" caused \\\"stat /run/host-services/docker.proxy.sock: no such file or directory\\\"\"": unknown
ERRO[0041] Failed to create cluster >>> Rolling Back
INFO[0041] Deleting cluster 'mycluster'
INFO[0041] Deleted k3d-mycluster-serverlb
INFO[0041] Deleted k3d-mycluster-server-0
INFO[0041] Deleted k3d-mycluster-tools
INFO[0041] Deleting cluster network 'k3d-mycluster'
INFO[0041] Deleting image volume 'k3d-mycluster-images'
FATA[0041] Cluster creation FAILED, all changes have been rolled back!

I have these installed and just upgraded with brew:

k3d version v5.0.1
docker engine version 19.03.13
docker containerd version v1.3.7
docker runc version 1.0.0-rc10
docker docker-init version 0.18.0

Do you maybe have a suggestion on what to do?

manumartin · 2021-10-24T21:12:55Z

At first I downgraded k3d to 4.x and that fixed the problem but k3d 5.x with the newest docker/containerd versions also fixed it for me.

franpog859 · 2021-10-25T06:37:08Z

Thanks, @manumartin ! Upgrading it with brew didn't work for me. Either something is wrong with linking the binary or there is not the latest version of Docker available here. Anyway, upgrading it just via GUI (sic!) worked fine :D

tiswanso · 2021-10-27T23:14:44Z

Hit the same issue on Ubuntu 18.0.4 with the same docker, containerd, and runc versions @franpog859 mentioned above.

Upgraded docker via apt upgrade docker-ce to resolve it. The k3d prereqs should mention version compatibility or min version requirements.

The resulting versions that worked for me:

Client: Docker Engine - Community
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        b485636
 Built:             Mon Oct 25 07:42:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.10
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       e2f740d
  Built:            Mon Oct 25 07:41:06 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

mjpowersjr · 2021-11-10T19:53:00Z

I had the same exact problem, 4.x worked, upgrade to 5.x broke. As @tiswanso and others suggest, upgrading Docker fixed the issue. In case it helps - this is the version of Docker I was using on Ubuntu when in a broken state:

Docker version 20.10.5, build 55c4c88

iwilltry42 · 2021-11-11T07:21:29Z

Hi @manumartin , thanks for opening this issue and thanks to the others for providing additional input and a "fix"/workaround.
The last update related to Docker in the tools package was a dependency update from docker v20.10.5 to v20.10.8 which already landed in k3d v4.4.8 in August 🤔 (#679) (similar for k3d itself)
I'm not sure if we can fix forward here, but I'll investigate today 👍

iwilltry42 · 2021-11-11T09:24:34Z

Tested some versions of Docker now with k3d v5.1.0 and I can confirm that it only starts working with Docker v20.10.4+ 🤔

tiswanso · 2021-11-12T15:46:52Z

@iwilltry42 -- in my case just documenting the version dependencies would have spared me a decent amount of effort. I'm just using k3d in an environment I completely control so updating dependencies is not much of a problem. Ideally, the k3d doc has a version compatibility matrix.

iwilltry42 · 2021-11-12T15:54:28Z

@tiswanso , I'm definitely going to add a requirement for Docker >=v20.10.5.
Unfortunately, there are just 5 million things to consider in compatibility matrices: OS (Linux/Mac/Windows) x Docker Variant (plain docker, Docker Machine, Docker for Mac, Docker for Windows, Docker for Windows with WSL2 backend) x Docker Version (what we face now) x k3d version (including dependency versions) x K3s version x Kernel version.

k3d x K3s I already had in CI/CD, which broke CI due to taking way to long to go through the E2E Tests with all K3s versions.
I guess Docker versions we can cover with a simple test case that should be run before merging a dependency update though ✔️

Update: After some investigation, I feel like it's caused by runc (seems like the update v1.0.0-rc92...v1.0.0-rc93 fixed it)

Update 2: Updated requirements on k3d.io and in the Readme: https://k3d.io/v5.1.0/#requirements

schnatterer · 2022-01-11T15:03:26Z

Thanks for this issue, it saved me from a lot of debugging!

I can report an occurrence of this issue on docker 20.10.5 running on Ubuntu 20.04.2 LTS, which seems to have been shipped with runc 1.0.0-rc92 😬

@iwilltry42 wouldn't it make more sense to clarify the requirement to "a docker version that uses runc >= v1.0.0-rc93?

$ docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:20 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:15 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

iwilltry42 · 2022-01-12T20:18:05Z

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest.
I'll update the notes accordingly to reflect this.
Update: e82810a

schnatterer · 2022-01-12T23:17:31Z

@iwilltry42 yeah, I was wondering about this myself. Thanks for updating.

Following Arnold's recent recommendations, we've upgraded k3d to the latest 4.x release as the 5.x+ releases are not compatible with installed `runc` version. For more information please refer to: - openfun/arnold#733 - k3d-io/k3d#807 test docker executor rollback to the current ubuntu image Use the same circle version as Arnold Setup remote docker Upgrade k3d/kubectl releases restore machine executor Restore comment

regunakyle · 2022-04-24T13:24:33Z

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest. I'll update the notes accordingly to reflect this. Update: e82810a

Sorry for bumping.

Do I need to check the version of both Docker and runc, or just runc?
My Synology NAS has:

Docker 20.10.3
runc v1.0.0-rc93
containerd v1.4.3

Not sure if it can run K3d.

iwilltry42 · 2022-04-24T14:17:42Z

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest. I'll update the notes accordingly to reflect this. Update: e82810a

Sorry for bumping.

Do I need to check the version of both Docker and runc, or just runc?
My Synology NAS has:

Docker 20.10.3

runc v1.0.0-rc93

containerd v1.4.3

Not sure if it can run K3d.

Hmm.. as far as I could tell, runc was the issue here, so that should be enough.
I'm not certain though, so better just try 🤔

A long time has passed and the reason for this was: k3d-io/k3d#807 We should be able to use the builtin runc instead Signed-off-by: Charly Molter <[email protected]>

) A long time has passed and the reason for this was: k3d-io/k3d#807 We should be able to use the builtin runc instead --------- Signed-off-by: Charly Molter <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

manumartin added the bug Something isn't working label Oct 18, 2021

iwilltry42 self-assigned this Nov 11, 2021

iwilltry42 added this to the v5.1.1 milestone Nov 11, 2021

iwilltry42 added the docs Documentation label Nov 12, 2021

iwilltry42 closed this as completed Nov 14, 2021

iwilltry42 pinned this issue Nov 14, 2021

iwilltry42 changed the title ~~[BUG] k3d-tools container fails to start when trying to mount /var/run/docker.sock within the /var/run tmpfs mount~~ Docker >=v20.10.5 is required for k3d v5.x Nov 14, 2021

iwilltry42 changed the title ~~Docker >=v20.10.5 is required for k3d v5.x~~ ❗ Docker >=v20.10.5 is required for k3d v5.x ❗ Nov 14, 2021

iwilltry42 changed the title ~~❗ Docker >=v20.10.5 is required for k3d v5.x ❗~~ ❗ Docker >=v20.10.5 is required for k3d v5.x ❗ Nov 14, 2021

iwilltry42 changed the title ~~❗ Docker >=v20.10.5 is required for k3d v5.x ❗~~ ❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ Jan 12, 2022

shiddy mentioned this issue Feb 18, 2022

[BUG] k3d v5 client, create a new cluster: error "Failed to patch CoreDNS ConfigMap", cli rolls back and deletes cluster #779

Closed

AaronCrawfis mentioned this issue Mar 2, 2022

k3d doesn't work on M1 mac radius-project/radius#2003

Closed

jmaupetit mentioned this issue Oct 7, 2022

⬆️(project) upgrade k3d release to v5.4.6 openfun/arnold#756

Merged

1 task

lahabana mentioned this issue Feb 8, 2023

ci(k3d): update k3d to latest and remove runc install in circleci kumahq/kuma#5950

Merged

12 tasks

iwilltry42 unpinned this issue Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ #807

❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ #807

manumartin commented Oct 18, 2021 •

edited by iwilltry42

Loading

manumartin commented Oct 20, 2021

franpog859 commented Oct 24, 2021

manumartin commented Oct 24, 2021

franpog859 commented Oct 25, 2021

tiswanso commented Oct 27, 2021

mjpowersjr commented Nov 10, 2021

iwilltry42 commented Nov 11, 2021 •

edited

Loading

iwilltry42 commented Nov 11, 2021 •

edited

Loading

tiswanso commented Nov 12, 2021

iwilltry42 commented Nov 12, 2021 •

edited

Loading

schnatterer commented Jan 11, 2022

iwilltry42 commented Jan 12, 2022 •

edited

Loading

schnatterer commented Jan 12, 2022

regunakyle commented Apr 24, 2022

iwilltry42 commented Apr 24, 2022

❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ #807

❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ #807

Comments

manumartin commented Oct 18, 2021 • edited by iwilltry42 Loading

What did you do

What did you expect to happen

Screenshots or terminal output

Which OS & Architecture

Which version of k3d

Which version of docker

manumartin commented Oct 20, 2021

franpog859 commented Oct 24, 2021

manumartin commented Oct 24, 2021

franpog859 commented Oct 25, 2021

tiswanso commented Oct 27, 2021

mjpowersjr commented Nov 10, 2021

iwilltry42 commented Nov 11, 2021 • edited Loading

iwilltry42 commented Nov 11, 2021 • edited Loading

tiswanso commented Nov 12, 2021

iwilltry42 commented Nov 12, 2021 • edited Loading

schnatterer commented Jan 11, 2022

iwilltry42 commented Jan 12, 2022 • edited Loading

schnatterer commented Jan 12, 2022

regunakyle commented Apr 24, 2022

iwilltry42 commented Apr 24, 2022

manumartin commented Oct 18, 2021 •

edited by iwilltry42

Loading

Which version of `k3d`

iwilltry42 commented Nov 11, 2021 •

edited

Loading

iwilltry42 commented Nov 11, 2021 •

edited

Loading

iwilltry42 commented Nov 12, 2021 •

edited

Loading

iwilltry42 commented Jan 12, 2022 •

edited

Loading