Unable to create multiple cluster using kind 0.20.0+ on RedHat 8 with cgroup v1/cgroupns #3558
Comments
|
This didn't seem to take the first time-- the logs from when I did the above process |
|
Thanks for the detailed post, to be clear: I'm not able to use Azure at work, the Kubernetes project does not have funding from Azure currently, and I'm not going to be paying Azure personally (especially to debug a commercial non-open linux distro). I am however looking at the logs you uploaded. This bit from kubelet logs:
Looking around a bit more:
Unfortunately it's not too surprising that a 6 year old kernel is giving issues, I don't think any of the rest of the container ecosystem (IE docker, runc, containerd, kubernetes, ....) is actively testing on anything like this. It's difficult for us to extend better coverage than the underlying ecosystem. I'm guessing we are hitting issues with cgroupns=private with 4.18 You could try kind v0.19 as a stopgap to confirm if it's cgroupns related. |
|
I completely understand your feelings about RedHat, given the recent rise in their evilness. It remains a popular distro, especially among larger companies the likes of which employ folks learning about Kubernetes. If it helps, I noted this problem on Rocky Linux 8, so it's not specific to the commercial version of RedHat. Azure is not required to replicate the issue, but is a convenient way to do so. It should be relatively easy to replicate using a local Rocky Linux 8 VM. So far my workaround has been to use Ubuntu. :-) |
|
Rephrasing a bit: I really appreciate how thorough this bug report is including the reproducing guidance! Just trying to be clear and realistic about prioritization for my own time. Ubuntu is a good workaround, the ecosystem has pretty solid coverage there (also Fedora in KIND at least). It's really strange that this is only the second cluster and not one of the common resource exhaustion failure modes like inotify limits. Could you grab the logs from the first cluster just to confirm the difference in serial log etc? |
|
Here are both sets of logs from a repeated repro session using Rocky Linux 8. I also added the system log from /var/log/messages: Repro step change to VM_IMAGE to use Rocky Linux 8 instead: # Based on https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-cli
export RESOURCE_GROUP_NAME=rg-sbonds-kind-cgroups
export VNET_NAME=vnet-sbonds-kind-cgroups
export LOCATION=westus2
export VM_NAME=sbonds-kind-cgroups-bug-demo
export VM_IMAGE=erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:latest
export ADMIN_USERNAME=sbonds_admin
# Generate an SSH key
ssh-keygen -t rsa -b 2048 -N "" -C sbonds-kind-cgroups -f "$HOME/.ssh/id_rsa_sbonds-kind-cgroups"
export SSH_PUBLIC_KEY_CONTENT=$(cat "$HOME/.ssh/id_rsa_sbonds-kind-cgroups.pub")
# Create resource group to hold VM and virtual network (vnet)
az group create \
--name "$RESOURCE_GROUP_NAME" \
--location "$LOCATION"
# Create VM with access provided by a local ssh key
az vm create \
--resource-group "$RESOURCE_GROUP_NAME" \
--name "$VM_NAME" \
--image "$VM_IMAGE" \
--admin-username "$ADMIN_USERNAME" \
--ssh-key-values "$SSH_PUBLIC_KEY_CONTENT" \
--size Standard_B2s \
--public-ip-sku Basic |
|
The second cluster adds fine on Rocky Linux 9. |
|
And v19 works fine on Rock Linux 8. Should you want the logs for comparison, here they are: |
|
Given that the old Kind works on the old RedHat and the new kind works on the new RedHat, this seems like a low priority to fix. Perhaps a documentation update and compatibility check could be added to make it easier to tell that Kind v20+ is known to have issues on RedHat/Rocky 8. |
|
In a related note: kubernetes/enhancements#4572 there is open discussion to drop cgroup v1 support in Kubernetes. Similarly, systemd is going to drop v1, once that filters into our base image (and we'll want to keep getting systemd patches) we'll have the same issue there systemd/systemd#30852 Hard to say exact timeline, but the ecosystem around us is moving away from supporting v1 and that will start to make attempts for kind to support v1 moot, making it harder to prioritize doubling back to dig deeper on this case. Per Kubernetes's own docs, a 5.8+ kernel will be needed for cgroup v2. Some time after this happens we will probably add a blanket check for docker/podman/nerdctl to be on a cgroup v2 host, which will be a lot less brittle than attempting to detect the set of distros with broken v1+cgroupns (also not sure if docker etc expose enough information there but they do for v1/v2, inspecting the host local to |
|
@sbonds did you test it with cgroup v2 enabled? Its supported in EL8! |
No, but it sounds like a plausible workaround. In my case, it was easier to simply switch to Ubuntu. |
|
I am encountering a similar-sounding error while trying to update Kind (from a rather old version) running inside a GKE pod with DinD. The kernel in this case is 6.1.85 (Container-Optimized OS from Google, containerd://1.7.15, v1.28.11-gke.1260000); the DinD pod is running Ubuntu Jammy, in case it matters. Would you be interested in precise steps to reproduce or would the recommendation remain that you just cannot do this with Kind anyway? |
Kubernetes CI uses this combination, @BenTheElder had an issue about this |
|
We are running on Ubuntu, not COS. (Previously we needed this to get ipv6 kernel modules for local testing, and we have not switched back) |
This bug is nearly identical to #3340 which was thought to be specific to Amazon Linux, but this appears on RedHat 8 and Rocky Linux 8.
What happened:
Creating a second cluster fails. The message seen from the kind command is:
The Docker logs show more info such as:
What you expected to happen:
Creating a second cluster succeeds.
How to reproduce it (as minimally and precisely as possible):
Well, I figure you want precisely more than minimally. These steps may also be found at https://pastebin.com/RjGt0RBN. Apologies for the ads they will insert.
Reproduce on a RedHat Azure VM
You will need an Azure account and be logged in with the ability to create resource groups and VMs (e.g. "Contributor".) Running this test will cost between $0.25 and $1.00 provided you move at a reasonable pace.
Running these commands can be done from multiple possible places, but one of the most consistent is to use the Microsoft Azure CLI Docker container:
docker run -i -t mcr.microsoft.com/azure-cliThe only real annoyance about the Microsoft container is it doesn't have ssh installed. It can be installed with:
apk add opensshFrom a bash shell run this, adjusting names as needed. This should work as-is if you don't want to change away from the "sbonds" names:
Log in to the new VM
Put the public IP returned from the above "az vm create" command into
VM_IP:SSH to the public IP address using the SSH private key and the admin username:
Install Docker
To become root:
To not be root, type "exit" and it leaves the sudo session, reverting back to your original login.
Docker installation
As root:
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo dnf -y install docker-ce docker-ce-cli containerd.io systemctl start docker systemctl enable docker docker image ls usermod -G sbonds_admin,wheel,docker sbonds_adminTest from non-root ("sbonds") account. No images yet, so this is empty:
Install Kind
As root:
Log out and back in
Log out and back in again to pick up the results of the "usermod" which allows non-root access to Docker.
Up-arrow will restore your prior ssh command:
Create default kind cluster
Output:
Create non-default kind cluster
kind create cluster --kubeconfig $HOME/.kube/kubeconfig_nondefault-cluster --name nondefault-cluster --retainCollect logs
Copy the logs to your local location
Exit the remote ssh connection
exitscp the logs locally
Also create a tar archive if you'll need to copy the files out of the Docker container.
If necessary, copy it out of your Docker container:
From a local command line (bash, Powershell, etc.):
Find the ID of your
mcr.microsoft.com/azure-cliimage and usedocker cpto copy out thekind-logs.tar.gzfile created earlier.docker cp bbede2b3ace1:/root/kind-logs.tar.gz .7-zip and other utilities can handle .tar.gz files.
Destroy your test environment
This stops those Azure charges from accumulating. Be sure to get the resource group name right since this command is extremely destructive and does not prompt for confirmation.
Last chance to stop! Be sure this is correct since the next command can do a lot of damage quickly if the wrong group is used.
az group delete --resource-group "$RESOURCE_GROUP_NAME" --yesAnything else we need to know?:
Environment:
kind version): v0.22.0docker info,podman infoornerdctl info): Docker 26.0.0/etc/os-release): RHEL 8.9kubectl version): didn't install kubectlTasks
The text was updated successfully, but these errors were encountered: