Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nodes with custom domain-name fail to join EKS cluster on Kubernetes 1.26 #3028

Closed
FernandoMiguel opened this issue Apr 18, 2023 · 21 comments · Fixed by #3033
Closed

Nodes with custom domain-name fail to join EKS cluster on Kubernetes 1.26 #3028

FernandoMiguel opened this issue Apr 18, 2023 · 21 comments · Fixed by #3033
Assignees
Labels
status/needs-triage Pending triage or re-evaluation type/bug Something isn't working

Comments

@FernandoMiguel
Copy link

FernandoMiguel commented Apr 18, 2023

Image I'm using:
1.13.3-752a994d

What I expected to happen:
brand new EKS 1.26 cluster, to have MNG join the cluster

What actually happened:
I just created a new test cluster with eks 1.26.
ASG for MNG picked the following AMI
bottlerocket-aws-k8s-1.26-x86_64-v1.13.3-752a994d

these two nodes failed to join the cluster
image

How to reproduce the problem:

these are our subnet settings
image

bash-5.1# apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "752a994d",
    "pretty_name": "Bottlerocket OS 1.13.3 (aws-k8s-1.26)",
    "variant_id": "aws-k8s-1.26",
    "version_id": "1.13.3"
  }
}
bash-5.1# systemctl status kubelet                                                                                                                                                                                                                                                                                                                                                           ● kubelet.service - Kubelet                        Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─dockershim-symlink.conf
             /etc/systemd/system/kubelet.service.d
             └─exec-start.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─load-ipvs-modules.conf, make-kubelet-dirs.conf, prestart-pull-pause-ctr.conf
     Active: active (running) since Tue 2023-04-18 15:58:36 UTC; 15min ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 1151 (kubelet)
      Tasks: 12 (limit: 4552)
     Memory: 181.3M
        CPU: 3.891s
     CGroup: /runtime.slice/kubelet.service
             └─ 1151 /usr/bin/kubelet --cloud-provider external --kubeconfig /etc/kubernetes/kubelet/kubeconfig --config /etc/kubernetes/kubelet/config --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --containerd=/run/containerd/containerd.sock --root-dir /var/lib/kubelet --cert-dir /var/lib/kubelet/pki --node-ip 10.24.24.75 --node-labels eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=fernando-xm-ng-20230418155728076400000001,eks.amazonaws.com/nodegroup-image=ami-01391b390209af9c4,eks.amazonaws.com/sourceLaunchTemplateId=lt-0409dd3cc95200c4b,eks.amazonaws.com/sourceLaunchTemplateVersion=1 --register-with-taints "" --pod-infra-container-image 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.1-eksbuild.1

Apr 18 16:13:45 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: I0418 16:13:45.631480    1151 kubelet_node_status.go:70] "Attempting to register node" node="i-078b5b5ea3aedf1cb.compute.internal"
Apr 18 16:13:45 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:45.635716    1151 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes \"i-078b5b5ea3aedf1cb.compute.internal\" is forbidden: node \"i-078b5b5ea3aedf1cb.ec2.internal\" is not allowed to modify node \"i-078b5b5ea3aedf1cb.compute.internal\"" node="i-078b5b5ea3aedf1cb.compute.internal"
Apr 18 16:13:46 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:46.310563    1151 eviction_manager.go:261] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"i-078b5b5ea3aedf1cb.compute.internal\" not found"
Apr 18 16:13:52 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:52.378553    1151 controller.go:146] failed to ensure lease exists, will retry in 7s, error: leases.coordination.k8s.io "i-078b5b5ea3aedf1cb.compute.internal" is forbidden: User "system:node:i-078b5b5ea3aedf1cb.ec2.internal" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Apr 18 16:13:52 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: I0418 16:13:52.636622    1151 kubelet_node_status.go:70] "Attempting to register node" node="i-078b5b5ea3aedf1cb.compute.internal"
Apr 18 16:13:52 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:52.640732    1151 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes \"i-078b5b5ea3aedf1cb.compute.internal\" is forbidden: node \"i-078b5b5ea3aedf1cb.ec2.internal\" is not allowed to modify node \"i-078b5b5ea3aedf1cb.compute.internal\"" node="i-078b5b5ea3aedf1cb.compute.internal"
Apr 18 16:13:56 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:56.311088    1151 eviction_manager.go:261] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"i-078b5b5ea3aedf1cb.compute.internal\" not found"
Apr 18 16:13:59 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:59.380634    1151 controller.go:146] failed to ensure lease exists, will retry in 7s, error: leases.coordination.k8s.io "i-078b5b5ea3aedf1cb.compute.internal" is forbidden: User "system:node:i-078b5b5ea3aedf1cb.ec2.internal" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Apr 18 16:13:59 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: I0418 16:13:59.642252    1151 kubelet_node_status.go:70] "Attempting to register node" node="i-078b5b5ea3aedf1cb.compute.internal"
Apr 18 16:13:59 i-078b5b5ea3aedf1cb.compute.internal kubelet[1151]: E0418 16:13:59.644935    1151 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes \"i-078b5b5ea3aedf1cb.compute.internal\" is forbidden: node \"i-078b5b5ea3aedf1cb.ec2.internal\" is not allowed to modify node \"i-078b5b5ea3aedf1cb.compute.internal\"" node="i-078b5b5ea3aedf1cb.compute.internal"
bash-5.1# apiclient get settings.kubernetes
{
  "settings": {
    "kubernetes": {
      "api-server": "https://XXXX.yl4.us-east-1.eks.amazonaws.com",
      "authentication-mode": "aws",
      "cloud-provider": "aws",
      "cluster-certificate": "<>",
      "cluster-dns-ip": "172.20.0.10",
      "cluster-domain": "cluster.local",
      "cluster-name": "fernando-xm",
      "max-pods": 29,
      "node-ip": "10.24.24.75",
      "node-labels": {
        "eks.amazonaws.com/capacityType": "ON_DEMAND",
        "eks.amazonaws.com/nodegroup": "fernando-xm-ng-20230418155728076400000001",
        "eks.amazonaws.com/nodegroup-image": "ami-01391b390209af9c4",
        "eks.amazonaws.com/sourceLaunchTemplateId": "lt-0409dd3cc95200c4b",
        "eks.amazonaws.com/sourceLaunchTemplateVersion": "1"
      },
      "pod-infra-container-image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.1-eksbuild.1",
      "provider-id": "aws:///us-east-1a/i-078b5b5ea3aedf1cb",
      "server-tls-bootstrap": true,
      "standalone-mode": false
    }
  }
}
bash-5.1# apiclient get settings.kubernetes.node-ip
{
  "settings": {
    "kubernetes": {
      "node-ip": "10.24.24.75"
    }
  }
}
bash-5.1# pluto node-ip
"10.24.24.75"

bash-5.1# netdog generate-hostname
"i-078b5b5ea3aedf1cb.compute.internal"
$ aws ec2 describe-instance-attribute --instance-id i-078b5b5ea3aedf1cb --attribute userData --query "UserData.Value" --output text | base64 -d
[settings.kubernetes]
"cluster-name" = "fernando-xm"
"api-server" = "https://XXXX.yl4.us-east-1.eks.amazonaws.com"
"cluster-certificate" = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EUXhPREV5TVRReE9Gb1hEVE16TURReE5URXlNVFF4T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTFh3CmFScXJFczhOL0RZZ2t6ejJMVldWdmN0ZWJmdEJTN0g0MCtGY1A2b2ZWYk92UVdjVStqZ0kwbEIwWW9LYTVneVgKZDZBQVdoRUNMcWZ0TnNQZFJBWkhGK3VSQncxcThITms3RUhGemhXNlk0Ym5Dci9ERDdHdFdvblIrb2pQMVNkRgpNeEdzYVQvN0YraUtPUk5iOG0rV0JlM3R0RWZ4NWx5YzkvS20wZHJyVVlPbnhvVWljSkQ5YnZ4a3p3RGxNVEFSCnAxdFJNK1RFbTF2aUdTMmo2S1pmR3JKbDByYmk2aGdtY1U2TW1XSWhCbDdYY1g4OXROdEhtQnpHcXZMSG1URmEKSzQ5WThPUXMvK1o2aVo2VFQ2MWZQeWZkVnZ1T24wT0g1V0h5c0NUTW1tRUZkbis5blV4R0s5K2lkODJnem9nYgp4M0pEekFDSWJBUTIwUEVkQ0I4Q0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNRkhHTHY0eDJvaHJCUERpbUJJT2o0dDZETWZNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSlg0Y2M4TGN3MmU3eVNyY011RgpjSEhVNDYrNzJSUDE4Q1VWaFBGdlhsNVcxVi91amxsYmlKOVVjTnJsdFhkZklJeC9RTjJDRytrbmhPWUR5bStyCmZqZGQ1czU5dHdKZ3hnU2k2ZjBzNGJaVnBjZmVtQlJlc1FTMVNORTh2dUZLU1ZEbnIyU2pRcG1JdFZEenlKa1oKc1pSa1pPb0M3ZG1iUDBuU3lnMllpeHlBQkQ5MVpuYVBtOXhXZHc5c3lBSnJqeG5GcTlvMEI4U1NLclZMbGlabwoza2trYjczQjB6YmFZSnNTWHVYTW13Z0kzSGxlZHU5ei9ZUWFFaVFSYnFHYWZ6TE1aYzErSGNITW1hWC9SOHorCjU0VnFrbGlKcmZSRE5CbkZjKzQvcytmZFIzcjJiTVhqR3F0R3dReFRFaXN4ZjVZbURucllDY3IydWpHZEZ6TTEKQUUwPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
"cluster-dns-ip" = "172.20.0.10"
"max-pods" = 29
[settings.kubernetes.node-labels]
"eks.amazonaws.com/sourceLaunchTemplateVersion" = "1"
"eks.amazonaws.com/nodegroup-image" = "ami-01391b390209af9c4"
"eks.amazonaws.com/capacityType" = "ON_DEMAND"
"eks.amazonaws.com/nodegroup" = "fernando-xm-ng-20230418155728076400000001"
"eks.amazonaws.com/sourceLaunchTemplateId" = "lt-0409dd3cc95200c4b"
@FernandoMiguel FernandoMiguel added status/needs-triage Pending triage or re-evaluation type/bug Something isn't working labels Apr 18, 2023
@bcressey
Copy link
Contributor

Can you provide the PrivateDNSName field reported by EC2 API?

$ aws ec2 describe-instances --instance-ids <instance-id> | jq -r '.Reservations[].Instances[].PrivateDnsName'

@etungsten
Copy link
Contributor

etungsten commented Apr 18, 2023

Hi @FernandoMiguel,

It seems like Bottlerocket is getting the instance hostname i-<id>.compute.internal from IMDS, when the aws cloud provider is expecting i-<id>.ec2.internal. Looking at the docs for instance hostnames, us-east-1 is special and suffixes with .ec2.internal.

On the host, can you query IMDS for me to check what it returns for the hostname?
If IMDSv2:
local-hostname:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-hostname

hostname:

curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/hostname

@FernandoMiguel
Copy link
Author

@bcressey

 aws ec2 describe-instances --instance-ids i-078b5b5ea3aedf1cb | jq -r '.Reservations[].Instances[].PrivateDnsName'
i-078b5b5ea3aedf1cb.ec2.internal

@FernandoMiguel
Copy link
Author

@etungsten

[ssm-user@control]$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0  49079      0 --:--:-- --:--:-- --:--:-- 56000
[ssm-user@control]$ curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-hostname
i-078b5b5ea3aedf1cb.compute.internal

[ssm-user@control]$ curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/hostname
i-078b5b5ea3aedf1cb.compute.internal

@etungsten
Copy link
Contributor

etungsten commented Apr 18, 2023

Hi @FernandoMiguel,

We're still looking into the issue. There is a discrepancy specifically in us-east-1 with what IMDS returns for the hostname and what the EC2 API returns for PrivateDnsName which is what the AWS Cloud Provider uses for the node name.

As a temporary workaround, you can try setting the settings.network.hostname in Bottlerocket via a boostrap container. Here is an example of a bootstrap container that sets up the settings.kuberenets.max-pods setting: #1692

The bash script in the bootstrap container could do something like this:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
instance_id=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)

# Set the hostname directly
apiclient set network.hostname="${instance_id}.ec2.internal"

Please let us know if you have any questions about setting this up.

@FernandoMiguel
Copy link
Author

We aren't in a super hurry to upgrade.
I just been testing on the day it came out.
I can wait for a little while longer for the bottlerocket team to fix this in newer images..

If it helps, I can spin up a cluster in other regions besides us-east-1.

@etungsten etungsten changed the title Nodes fail to join EKS cluster on Kubernetes 1.26 (follow up from #3013) us-east-1 nodes and instances with custom domain-name fail to join EKS cluster on Kubernetes 1.26 Apr 18, 2023
@etungsten etungsten changed the title us-east-1 nodes and instances with custom domain-name fail to join EKS cluster on Kubernetes 1.26 us-east-1 nodes and nodes with custom domain-name fail to join EKS cluster on Kubernetes 1.26 Apr 18, 2023
@etungsten
Copy link
Contributor

If it helps, I can spin up a cluster in other regions besides us-east-1.

Yeah, try that out if you'd like. If our assessment is correct, the nodes should be able to join the cluster as long as it's not in us-east-1 or using custom domain in the DHCP options.

@etungsten
Copy link
Contributor

Hmm, I just created a new EKS cluster in us-east-1 with bottlerocket nodes and they all joined the cluster with the correct node names:

$ kubectl get nodes -o wide
NAME                            STATUS   ROLES    AGE   VERSION               INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                                KERNEL-VERSION   CONTAINER-RUNTIME
ip-192-168-3-68.ec2.internal    Ready    <none>   11m   v1.26.2-eks-b106822   192.168.3.68    44.200.125.4    Bottlerocket OS 1.13.3 (aws-k8s-1.26)   5.15.102         containerd://1.6.19+bottlerocket
ip-192-168-49-66.ec2.internal   Ready    <none>   10m   v1.26.2-eks-b106822   192.168.49.66   52.55.234.191   Bottlerocket OS 1.13.3 (aws-k8s-1.26)   5.15.102         containerd://1.6.19+bottlerocket

So there must be something different that's causing IMDS to return the incorrect hostname.

In any case, #3033 should fix the issue you're seeing @FernandoMiguel.

@FernandoMiguel
Copy link
Author

@etungsten what else can I provide to help understand what is causing the issue so you can reproduce?

@etungsten
Copy link
Contributor

Hi @FernandoMiguel,

It would be good to verify your cluster VPC DNS and DHCP options.

The VPC DNS settings:
image

Do the DHCP options look like this?
image

@FernandoMiguel
Copy link
Author

FernandoMiguel commented Apr 19, 2023

image

@etungsten

@etungsten
Copy link
Contributor

etungsten commented Apr 19, 2023

It seems like the domain-name for your us-east-1 VPC DHCP isn't matching up with what EC2 thinks should be the private DNS names for your nodes. By default, the domain name for us-east-1 clusters should be set to ec2.internal if I'm understanding this correctly. This essentially becomes the same issue with custom domain-names.

Do you know if terraform had somehow overridden that through https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc_dhcp_options#domain_name?

Another workaround you can use for now (before #3033 gets released in a new version) is to change the domain name in your DHCP option to ec2.internal and your nodes should be able to join the cluster.

@etungsten etungsten changed the title us-east-1 nodes and nodes with custom domain-name fail to join EKS cluster on Kubernetes 1.26 Nodes with custom domain-name fail to join EKS cluster on Kubernetes 1.26 Apr 19, 2023
@FernandoMiguel
Copy link
Author

I'm not familiar with any particular changes done on our side, and this is a relatively new account.
I'll check tomorrow the code for the VPC creation, when I get back in the office.

@alicancakil
Copy link

alicancakil commented Apr 19, 2023

I started to experience the same issue after 1.26 upgrade with Bottlerocket OS 1.13.3

I am in ca-central-1 region, and I use AWS DHCP to hand out custom domains.
Everything was functioning properly with 1.24. We upgraded to 1.25 briefly and then went to 1.26

  • Based on kubelet logs confusion is caused here;

ip-10-10-165-215.mydomain.com
v.s.
ip-10-10-165-215.ca-central-1.compute.internal

Apr 19 16:37:55 ip-10-10-165-215.mydomain.com kubelet[1156]: I0419 16:37:55.005865    1156 kubelet_node_status.go:70] "Attempting to register node" node="ip-10-10-165-215.mydomain.com"
Apr 19 16:37:55 ip-10-10-165-215.mydomain.com kubelet[1156]: E0419 16:37:55.007234    1156 controller.go:146] failed to ensure lease exists, will retry in 400ms, error: leases.coordination.k8s.io "ip-10-10-165-215.mydomain.com" is forbidden: User "system:node:ip-10-10-165-215.ca-central-1.compute.internal" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node

@alicancakil
Copy link

Update: Rolling back to bottlerocket OS version 1.13.2 fixed the issue for me.

@mfeldheim
Copy link

mfeldheim commented Apr 20, 2023

If it helps, I can spin up a cluster in other regions besides us-east-1.

Yeah, try that out if you'd like. If our assessment is correct, the nodes should be able to join the cluster as long as it's not in us-east-1 or using custom domain in the DHCP options.

I have this issue in eu-west-1 with default hostname settings and DHCP option set (no domain name), tried to update DHCP option set domainname to ec2.internal, didn't make a difference, still a name mismatch for the registration call

I have noticed, that on 1.13.2, the node only has a single private IP whereas before it always had 2 (ipv4). That might be due the the failed cluster registration and no CNI interaction to that point

@mfeldheim
Copy link

mfeldheim commented Apr 20, 2023

have this issue in eu-west-1 with default hostname settings and DHCP option set (no domain name), tried to update DHCP option set domainname to ec2.internal, didn't make a difference, still a name mismatch for the registration call

OK, kubelet logs reveal the issue in my region. I just had a very old DHCP option set and eu-west-1.compute.internal is the correct domain to set for eu-west-1

Apr 20 20:32:35 ip-10-2-12-62.ec2.internal kubelet[1316]: E0420 20:32:35.906900    1316 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes \"ip-10-2-12-62.ec2.internal\" is forbidden: node \"ip-10-2-12-62.eu-west-1.compute.internal\" is not allowed to modify node \"ip-10-2-12-62.ec2.internal\"" node="ip-10-2-12-62.ec2.internal"

@FernandoMiguel
Copy link
Author

@etungsten once #3033 is merged and new AMI is available, what steps, if any, will we have to take to make this work with not changes to the VPC config?

@yeazelm
Copy link
Contributor

yeazelm commented Apr 21, 2023

@FernandoMiguel Once the AMIs are available with #3033, the behavior should be similar to before 1.26 where the hostname provided to kubelet is the one expected by the cluster. This shouldn't require any changes on the VPC side.

@FernandoMiguel
Copy link
Author

great news. eagerly waiting for its release.

@FernandoMiguel
Copy link
Author

just spun up a new test cluster.
AMI picked was
│ OS Image: Bottlerocket OS 1.13.4 (aws-k8s-1.26)
I can confirm my issue is now fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/needs-triage Pending triage or re-evaluation type/bug Something isn't working
Projects
None yet
6 participants