Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

calico-kube-controllers cannot create resource "tiers" #11942

Open
3v01ut10n opened this issue Feb 3, 2025 · 0 comments
Open

calico-kube-controllers cannot create resource "tiers" #11942

3v01ut10n opened this issue Feb 3, 2025 · 0 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@3v01ut10n
Copy link

3v01ut10n commented Feb 3, 2025 •
edited
Loading

What happened?

After upgrading cluster 1.30.4 -> 1.31.4 from version 2.26.0 to 2.27.0 an error occurs in pod "calico-kube-controllers"

2025-02-03 13:15:10.012 [WARNING][1] kube-controllers/client.go 417: Unauthorized to create tier default. error=connection is unauthorized: tiers.crd.projectcalico.org is forbidden: User "system:serviceaccount:kube-system:calico-kube-controllers" cannot create resource "tiers" in API group "crd.projectcalico.org" at the cluster scope
2025-02-03 13:15:10.013 [WARNING][1] kube-controllers/client.go 417: Unauthorized to create tier adminnetworkpolicy. error=connection is unauthorized: tiers.crd.projectcalico.org is forbidden: User "system:serviceaccount:kube-system:calico-kube-controllers" cannot create resource "tiers" in API group "crd.projectcalico.org" at the cluster scope

What did you expect to happen?

After launch there should be a clean output in the log

I0203 13:36:16.704538       1 shared_informer.go:311] Waiting for caches to sync for nodes
I0203 13:36:16.805355       1 shared_informer.go:318] Caches are synced for nodes
I0203 13:36:16.805387       1 shared_informer.go:311] Waiting for caches to sync for pods
I0203 13:36:16.805394       1 shared_informer.go:318] Caches are synced for pods

How can we reproduce it (as minimally and precisely as possible)?

Run kubespray with calico and check log calico-kube-controllers

OS

Linux 5.15.0-125-generic x86_64
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version of Ansible

ansible [core 2.16.14]
  config file = /home/***/kubespray/ansible.cfg
  configured module search path = ['/home/***/kubespray/library']
  ansible python module location = /home/***/kubespray/venv/lib/python3.10/site-packages/ansible
  ansible collection location = /home/***/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/***/kubespray/venv/bin/ansible
  python version = 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (/home/***/kubespray/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Version of Python

Python 3.10.12

Version of Kubespray (commit)

https://github.com/kubernetes-sigs/kubespray/releases/tag/v2.27.0

Network plugin used

calico

Full inventory with variables

Spoiler

node1 | SUCCESS => {
"hostvars[inventory_hostname]": {
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home//kubespray/ansible.cfg",
"ansible_connection": "local",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_inventory_sources": [
"/home//kubespray/inventory/test/hosts.ini"
],
"ansible_playbook_python": "/home//kubespray/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.14",
"major": 2,
"minor": 16,
"revision": 14,
"string": "2.16.14"
},
"apiserver_loadbalancer_domain_name": "elbtest..pw",
"bin_dir": "/usr/local/bin",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"group_names": [
"ungrouped"
],
"groups": {
"all": [
"node1",
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"etcd": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"k8s_cluster": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"kube_control_plane": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"kube_node": [
"k8s-test-worker-01"
],
"ungrouped": [
"node1"
]
},
"inventory_dir": "/home//kubespray/inventory/test",
"inventory_file": "/home//kubespray/inventory/test/hosts.ini",
"inventory_hostname": "node1",
"inventory_hostname_short": "node1",
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"loadbalancer_apiserver": {
"address": "10.9.2.51",
"port": 6442
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_release_dir": "{{ansible_env.HOME}}/releases",
"no_proxy_exclude_workers": false,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__3a9855b918751a62addc658864e7fc1b79128282",
"playbook_dir": "/home//kubespray",
"unsafe_show_logs": false
}
}
k8s-test-master-03 | SUCCESS => {
"hostvars[inventory_hostname]": {
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home//kubespray/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.9.2.53",
"ansible_inventory_sources": [
"/home//kubespray/inventory/test/hosts.ini"
],
"ansible_playbook_python": "/home//kubespray/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.14",
"major": 2,
"minor": 16,
"revision": 14,
"string": "2.16.14"
},
"apiserver_loadbalancer_domain_name": "elbtest..pw",
"argocd_enabled": false,
"auto_renew_certificates": true,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_ipip_mode": "CrossSubnet",
"calico_network_backend": "bird",
"calico_pool_blocksize": 26,
"calico_vxlan_mode": "Never",
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home//kubespray/inventory/test/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane"
],
"groups": {
"all": [
"node1",
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"etcd": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"k8s_cluster": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"kube_control_plane": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"kube_node": [
"k8s-test-worker-01"
],
"ungrouped": [
"node1"
]
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home//kubespray/inventory/test",
"inventory_file": "/home//kubespray/inventory/test/hosts.ini",
"inventory_hostname": "k8s-test-master-03",
"inventory_hostname_short": "k8s-test-master-03",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": true,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.31.4",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "89b8438aad0bbacd56122e7ebb8ee9da4a7f0d1b9bb2c4a9d5e92f93a1d6ef5f",
"kubeadm_patches": [],
"kubeadm_patches_dir": "/etc/kubernetes/patches",
"kubelet_image_gc_high_threshold": 85,
"kubelet_image_gc_low_threshold": 80,
"kubelet_max_pods": 160,
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.9.2.51",
"port": 6442
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_auto_assign": true,
"metallb_avoid_buggy_ips": true,
"metallb_config": {
"address_pools": {
"loadbalanced": {
"auto_assign": true,
"ip_range": [
"10.9.2.251/32"
]
}
},
"layer2": [
"loadbalanced"
]
},
"metallb_enabled": true,
"metallb_namespace": "metallb-system",
"metallb_protocol": "layer2",
"metallb_speaker_enabled": true,
"metrics_server_enabled": true,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__3a9855b918751a62addc658864e7fc1b79128282",
"persistent_volumes_enabled": false,
"playbook_dir": "/home//kubespray",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
k8s-test-master-01 | SUCCESS => {
"hostvars[inventory_hostname]": {
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home//kubespray/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.9.2.51",
"ansible_inventory_sources": [
"/home//kubespray/inventory/test/hosts.ini"
],
"ansible_playbook_python": "/home//kubespray/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.14",
"major": 2,
"minor": 16,
"revision": 14,
"string": "2.16.14"
},
"apiserver_loadbalancer_domain_name": "elbtest..pw",
"argocd_enabled": false,
"auto_renew_certificates": true,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_ipip_mode": "CrossSubnet",
"calico_network_backend": "bird",
"calico_pool_blocksize": 26,
"calico_vxlan_mode": "Never",
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home//kubespray/inventory/test/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane"
],
"groups": {
"all": [
"node1",
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"etcd": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"k8s_cluster": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"kube_control_plane": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"kube_node": [
"k8s-test-worker-01"
],
"ungrouped": [
"node1"
]
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home//kubespray/inventory/test",
"inventory_file": "/home//kubespray/inventory/test/hosts.ini",
"inventory_hostname": "k8s-test-master-01",
"inventory_hostname_short": "k8s-test-master-01",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": true,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.31.4",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "89b8438aad0bbacd56122e7ebb8ee9da4a7f0d1b9bb2c4a9d5e92f93a1d6ef5f",
"kubeadm_patches": [],
"kubeadm_patches_dir": "/etc/kubernetes/patches",
"kubelet_image_gc_high_threshold": 85,
"kubelet_image_gc_low_threshold": 80,
"kubelet_max_pods": 160,
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.9.2.51",
"port": 6442
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_auto_assign": true,
"metallb_avoid_buggy_ips": true,
"metallb_config": {
"address_pools": {
"loadbalanced": {
"auto_assign": true,
"ip_range": [
"10.9.2.251/32"
]
}
},
"layer2": [
"loadbalanced"
]
},
"metallb_enabled": true,
"metallb_namespace": "metallb-system",
"metallb_protocol": "layer2",
"metallb_speaker_enabled": true,
"metrics_server_enabled": true,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__3a9855b918751a62addc658864e7fc1b79128282",
"persistent_volumes_enabled": false,
"playbook_dir": "/home//kubespray",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
k8s-test-master-02 | SUCCESS => {
"hostvars[inventory_hostname]": {
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home//kubespray/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.9.2.52",
"ansible_inventory_sources": [
"/home//kubespray/inventory/test/hosts.ini"
],
"ansible_playbook_python": "/home//kubespray/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.14",
"major": 2,
"minor": 16,
"revision": 14,
"string": "2.16.14"
},
"apiserver_loadbalancer_domain_name": "elbtest..pw",
"argocd_enabled": false,
"auto_renew_certificates": true,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_ipip_mode": "CrossSubnet",
"calico_network_backend": "bird",
"calico_pool_blocksize": 26,
"calico_vxlan_mode": "Never",
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home//kubespray/inventory/test/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane"
],
"groups": {
"all": [
"node1",
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"etcd": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"k8s_cluster": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"kube_control_plane": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"kube_node": [
"k8s-test-worker-01"
],
"ungrouped": [
"node1"
]
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home//kubespray/inventory/test",
"inventory_file": "/home//kubespray/inventory/test/hosts.ini",
"inventory_hostname": "k8s-test-master-02",
"inventory_hostname_short": "k8s-test-master-02",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": true,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.31.4",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "89b8438aad0bbacd56122e7ebb8ee9da4a7f0d1b9bb2c4a9d5e92f93a1d6ef5f",
"kubeadm_patches": [],
"kubeadm_patches_dir": "/etc/kubernetes/patches",
"kubelet_image_gc_high_threshold": 85,
"kubelet_image_gc_low_threshold": 80,
"kubelet_max_pods": 160,
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.9.2.51",
"port": 6442
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_auto_assign": true,
"metallb_avoid_buggy_ips": true,
"metallb_config": {
"address_pools": {
"loadbalanced": {
"auto_assign": true,
"ip_range": [
"10.9.2.251/32"
]
}
},
"layer2": [
"loadbalanced"
]
},
"metallb_enabled": true,
"metallb_namespace": "metallb-system",
"metallb_protocol": "layer2",
"metallb_speaker_enabled": true,
"metrics_server_enabled": true,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__3a9855b918751a62addc658864e7fc1b79128282",
"persistent_volumes_enabled": false,
"playbook_dir": "/home//kubespray",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
k8s-test-worker-01 | SUCCESS => {
"hostvars[inventory_hostname]": {
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home//kubespray/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.9.2.54",
"ansible_inventory_sources": [
"/home//kubespray/inventory/test/hosts.ini"
],
"ansible_playbook_python": "/home//kubespray/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.14",
"major": 2,
"minor": 16,
"revision": 14,
"string": "2.16.14"
},
"apiserver_loadbalancer_domain_name": "elbtest..pw",
"argocd_enabled": false,
"auto_renew_certificates": true,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_ipip_mode": "CrossSubnet",
"calico_network_backend": "bird",
"calico_pool_blocksize": 26,
"calico_vxlan_mode": "Never",
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home//kubespray/inventory/test/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"k8s_cluster",
"kube_node"
],
"groups": {
"all": [
"node1",
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"etcd": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"k8s_cluster": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03",
"k8s-test-worker-01"
],
"kube_control_plane": [
"k8s-test-master-01",
"k8s-test-master-02",
"k8s-test-master-03"
],
"kube_node": [
"k8s-test-worker-01"
],
"ungrouped": [
"node1"
]
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home//kubespray/inventory/test",
"inventory_file": "/home//kubespray/inventory/test/hosts.ini",
"inventory_hostname": "k8s-test-worker-01",
"inventory_hostname_short": "k8s-test-worker-01",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": true,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.31.4",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "89b8438aad0bbacd56122e7ebb8ee9da4a7f0d1b9bb2c4a9d5e92f93a1d6ef5f",
"kubeadm_patches": [],
"kubeadm_patches_dir": "/etc/kubernetes/patches",
"kubelet_image_gc_high_threshold": 85,
"kubelet_image_gc_low_threshold": 80,
"kubelet_max_pods": 160,
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.9.2.51",
"port": 6442
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_auto_assign": true,
"metallb_avoid_buggy_ips": true,
"metallb_config": {
"address_pools": {
"loadbalanced": {
"auto_assign": true,
"ip_range": [
"10.9.2.251/32"
]
}
},
"layer2": [
"loadbalanced"
]
},
"metallb_enabled": true,
"metallb_namespace": "metallb-system",
"metallb_protocol": "layer2",
"metallb_speaker_enabled": true,
"metrics_server_enabled": true,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__3a9855b918751a62addc658864e7fc1b79128282",
"persistent_volumes_enabled": false,
"playbook_dir": "/home/***/kubespray",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}

Command used to invoke ansible

ansible-playbook -i inventory/test/hosts.ini -b playbooks/facts.yml; ansible-playbook -i inventory/test/hosts.ini -b upgrade-cluster.yml --limit "kube_control_plane"; ansible-playbook -i inventory/test/hosts.ini -b upgrade-cluster.yml --limit "kube_node"

Output of ansible run

Anything else we need to know

This can be fixed in the file roles/kubernetes-apps/policy_controller/calico/templates/calico-kube-cr.yml.j2

Need to add this block

  # Create tiers.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - tiers
    verbs:
      - create

here:

{% if calico_datastore == "etcd"  %}
  - apiGroups:
    - ""
    - extensions
    resources:
      - pods
      - namespaces
      - networkpolicies
      - nodes
      - serviceaccounts
    verbs:
      - watch
      - list
      - get
  - apiGroups:
    - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
    - networking.k8s.io
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  # Create tiers.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - tiers
    verbs:
      - create
@3v01ut10n 3v01ut10n added the kind/bug Categorizes issue or PR as related to a bug. label Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@3v01ut10n