kubeadm bootstraps apiserver with mismatching IP famlies

[telmich]

What keywords did you search in kubeadm issues before filing this one?

• error creating bootstrap controller
• service IP family

There is #1612 however it does not seem to address this issue.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

[20:18] server121.place15:~# apk list -I | grep -e ^kube -e ^cri
cri-o-1.24.1-r2 x86_64 {cri-o} (Apache-2.0) [installed]
cri-o-openrc-1.24.1-r2 x86_64 {cri-o} (Apache-2.0) [installed]
kubeadm-1.24.4-r0 x86_64 {kubernetes} (Apache-2.0) [installed]
kubectl-1.24.4-r0 x86_64 {kubernetes} (Apache-2.0) [installed]
kubelet-1.24.4-r0 x86_64 {kubernetes} (Apache-2.0) [installed]
cri-tools-1.25.0-r0 x86_64 {cri-tools} (Apache-2.0) [installed]
kubelet-openrc-1.24.4-r0 x86_64 {kubernetes} (Apache-2.0) [installed]

[20:18] server121.place15:~# uname -a; cat /etc/alpine-release 
Linux server121 5.15.65-0-lts #1-Alpine SMP Tue, 06 Sep 2022 09:27:08 +0000 x86_64 Linux
3.15.0
[20:19] server121.place15:~# 

What happened?

Running apiserver fails with:

failed" err="error creating bootstrap controller: service IP family \"2a0a:e5c0:32:11::/108\" must match public address family \"212.103.65.231\""

What you expected to happen?

To bind to the correct IP family or to both

How to reproduce it (as minimally and precisely as possible)?

Bootstrap on a dual stack server with public IPv6 + public IPv4. I used the following config for kubeadm:

kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.24.4
networking:
  dnsDomain: server121.k8s.ooo
  podSubnet: 2a0a:e5c0:32:10::/64
  serviceSubnet: 2a0a:e5c0:32:11::/108
apiServer:
  extraArgs:
    service-node-port-range: 80-55000
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: cgroupfs
---

Anything else we need to know?

In a non k8s context I'd expect the api server to listen to all ip families. However if the host ip family is influenced by the pod sub net family, we might need to select the control plane endpoint depending on the podsubnet ip family

[telmich]

I tried to tell kubeadm to move the apiServer binding to IPv6 using the following configuration:

cat ~/kubeadm.yaml 
# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.24.4
networking:
  dnsDomain: server121.k8s.ooo
  podSubnet: 2a0a:e5c0:32:10::/64
  serviceSubnet: 2a0a:e5c0:32:11::/108
apiServer:
  extraArgs:
    service-node-port-range: 53-55000
controlPlaneEndpoint: "server121-api.k8s.ooo:6443"
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: cgroupfs
---

Note that controlPlaneEndpoint: "server121-api.k8s.ooo:6443" only resolves to IPv6 (only has AAAA) entries and no A entries. Thus the binding should only go to IPv6. However kubeadm generates the following incorrect manifest:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 212.103.65.231:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=212.103.65.231
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.server121.k8s.ooo
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=2a0a:e5c0:32:11::/108
    - --service-node-port-range=53-55000
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.24.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 212.103.65.231
        path: /livez
        port: 6443
        scheme: HTTPS
...

[telmich]

Correction: adding

kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
localAPIEndpoint:
  advertiseAddress: 2a00:c38::1:0:3e1f
  bindPort: 6443

to the configuration fixes the issue. However it would be great if kubeadm could detect that conflict by itself and bootstrap the correct api server parameters.

[neolit123]

advertiseAddress is autodetected from the default route on the host. if you'd like that ip to be v6 family, i think setting it explicitly is valid.

probing the control plane endpoint for ip family does not seem like something we want to do.

[neolit123]

please add further comments if needed.