[rootless] Create cluster

[moeryomenko]

What happened:
failed run control-plane node

I0325 08:47:15.247848     171 round_trippers.go:445] GET https://kind-external-load-balancer:6443/healthz?timeout=10s  in 1 milliseconds
I0325 08:47:15.247910     171 request.go:943] Got a Retry-After 1s response for attempt 5 to https://kind-external-load-balancer:6443/healthz?timeout=10s
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

	Unfortunately, an error has occurred:
		timed out waiting for the condition

	This error is likely caused by:
		- The kubelet is not running
		- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

	If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
		- 'systemctl status kubelet'
		- 'journalctl -xeu kubelet'

	Additionally, a control plane component may have crashed or exited when started by the container runtime.
	To troubleshoot, list all containers using your preferred container runtimes CLI.

	Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
		- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
		Once you have found the failing container, you can inspect its logs with:
		- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'

What you expected to happen:
running cluster

Environment:

• kind version: (use kind version):

kind v0.10.0 go1.16.1 linux/amd64

• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):

Client:
Context:    default
Debug Mode: false
Plugins:
 app: Docker App (Docker Inc., v0.9.1-beta3)
 buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 20.10.5
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 1
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
 seccomp
  Profile: default
Kernel Version: 5.11.8-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.454GiB
Name: yoga
ID: ZPTK:R2VF:JDYV:3CC6:CHHQ:BPMD:GE3Y:3C5V:OOVE:BW5X:ODMJ:5CWI
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: moeryomenko
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

• OS (e.g. from /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux

cluster-logs.zip

[BenTheElder]

Storage Driver: fuse-overlayfs

Is this rootless docker? Because Kubernetes does not support rootless yet. #1727 #1797
https://kind.sigs.k8s.io/docs/user/rootless/

Kind will support it in the next release. You can experiment with it by cloning the repo at HEAD and running make build then using the binary in bin/kind.

Will look at the rest of the logs later.

[BenTheElder]

Mar 25 09:27:44 kind-control-plane kubelet[580]: F0325 09:27:44.438723 580 kubelet.go:1350] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/lib/kubelet": could not find device with major: 259, minor: 2 in cached partitions map

that's the problem. usually this means means we need to mount /dev/mapper (can be manually mounted with a simple config until we automate it for this mode, we autodetect when /dev/mapper will be present/necessary and mount it usually)

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /dev/mapper
    containerPath: /dev/mapper

but also per docker's docs, fuse-overlayfs is only used for rootless mode, which is not supported in stable kind (will be supported in the next release) and not at all supported in kubernetes yet (we're working around this until then).

https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Note: For fuse-overlayfs driver, check Rootless mode documentation.

calling this a duplicate of #1797, you can try out rootless early by cloning the repo and running make build and then using the bin/kind binary (just needs make/posix shell).

[moeryomenko]

Storage Driver: fuse-overlayfs

Is this rootless docker? Because Kubernetes does not support rootless yet. #1727 #1797
https://kind.sigs.k8s.io/docs/user/rootless/

Kind will support it in the next release. You can experiment with it by cloning the repo at HEAD and running make build then using the binary in bin/kind.

Will look at the rest of the logs later.

Thank, cluster runned. But instead of docker, use podman

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.27-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 65fad4bfcb250df0435ea668017e643e7f462155'
  cpus: 4
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: yoga
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.9-arch1-1
  linkmode: dynamic
  memFree: 134897664
  memTotal: 3708649472
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 0.18-1
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /sbin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.9-1
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8191209472
  swapTotal: 8594124800
  uptime: 17m 11.01s
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/maxer/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /sbin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.5.0-1
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
  graphRoot: /home/maxer/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: f2fs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 42
  runRoot: /run/user/1000/containers
  volumePath: /home/maxer/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1613921386
  BuiltTime: Sun Feb 21 18:29:46 2021
  GitCommit: c640670e85c4aaaff92741691d6a854a90229d8d
  GoVersion: go1.16
  OsArch: linux/amd64
  Version: 3.0.1

logs.tar.gz