kubeadm init does not use the images downloaded by kubeadm config images pull

[tankilo]

What keywords did you search in kubeadm issues before filing this one?

If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:43:25Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): centos 7 run in vmware desktop vm
• Kernel (e.g. uname -a): Linux kmaster 3.10.0-1127.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Container runtime (CRI) (e.g. containerd, cri-o): containerd or cri-dockerd
• Container networking plugin (CNI) (e.g. Calico, Cilium):
• Others:

What happened?

I am in China, so i need to set --image-repository attribute.
First, i execute kubeadm config images pull to download images.

[vagrant@kmaster ~]$ kubeadm config images list --kubernetes-version 1.25.0
registry.k8s.io/kube-apiserver:v1.25.0
registry.k8s.io/kube-controller-manager:v1.25.0
registry.k8s.io/kube-scheduler:v1.25.0
registry.k8s.io/kube-proxy:v1.25.0
registry.k8s.io/pause:3.8
registry.k8s.io/etcd:3.5.4-0
registry.k8s.io/coredns/coredns:v1.9.3

[vagrant@kmaster ~]$ sudo kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers --kubernetes-version 1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.8
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.4-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.9.3

You can see, registry.aliyuncs.com/google_containers/pause:3.8 is downloaded.

Then i run sudo kubeadm init --apiserver-advertise-address=172.16.16.100 --kubernetes-version v1.25.0 --image-repository registry.aliyuncs.com/google_containers --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16

sudo journalctl -xeu kubelet

Aug 31 09:07:23 kmaster kubelet[10273]: E0831 09:07:23.996849   10273 remote_runtime.go:233] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.6\":
Aug 31 09:07:23 kmaster kubelet[10273]: E0831 09:07:23.996891   10273 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.6\": Error
Aug 31 09:07:23 kmaster kubelet[10273]: E0831 09:07:23.996948   10273 kuberuntime_manager.go:772] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.6\": Error
Aug 31 09:07:23 kmaster kubelet[10273]: E0831 09:07:23.997024   10273 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-scheduler-kmaster_kube-system(91310cf827a3e04a7a8fbefc4
Aug 31 09:07:24 kmaster kubelet[10273]: E0831 09:07:24.053871   10273 kubelet.go:2448] "Error getting node" err="node \"kmaster\" not found"

kubeadm init didn't use the image downloaded by command kubeadm config images pull!

What you expected to happen?

kubeadm init use the image downloaded by command kubeadm config images pull

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

[pacoxu]

check your containerd configurition for pause image and change it.

[neolit123]

yes, you need to separately set the pause image for the CR

/support

[github-actions]

Hello, @tankilo 🤖 👋

You seem to have troubles using Kubernetes and kubeadm.
Note that our issue trackers should not be used for providing support to users.
There are special channels for that purpose.

Please see:

• https://github.com/kubernetes/kubeadm#support

[neolit123]

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

[pacoxu]

This seems to be a very common problem for users with kubeadm + contained.

Should we add some tips for users init cluster with --image-repository?

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

The docs seems to be enough.

[neolit123]

we also mention it in the kubeadm init page
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-pause-image

[tankilo]

Pretty Thanks for the answer above, i didn' go through the document. I follow a blog written by others.

[neolit123]

official docs are the only up-to-date source and have all the details

[sujoshua]

According to the doc:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

It does work.

But it was set in the config and the pause image version depends on Kubernetes version, so it cannot be
permanently hardcoded.

That means every time I need to manually change the config in CR, it's not elegant. So do we have a better way to solve it?

[pacoxu]

@sujoshua

I am afraid there is no simple way.

For containerd in a future release(1.7+ maybe or 2.0?), containerd starts to use sandbox API instead of a pause container. See more at containerd/containerd#4131 (comment).

(BTW, For the pause image gc, there is a pinned image feature in https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2040-kubelet-cri#pinned-images. It can help for image not gc.)

[haji4ref]

check your containerd configurition for pause image and change it.

Thanks a lot! You saved me!