🌱 feat: adds new nodeadm bootstrap type #1

faiq · 2025-09-22T20:33:23Z

What type of PR is this?
/kind feature

Adds the new nodeadm type to provision EKS nodes that use nodeadm -- mostly used by AL2023 nodes. See KEP for more details and motivations.

The controller code largely resembles the existing code for EKSConfig reconciliation.

New unit tests and e2e tests were added to check logic. As well as a new e2e test to exercise upgrading.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

Release note:

ci: adds unit tests

bootstrap/eks/api/v1beta2/eksconfig_types.go

faiq · 2025-09-23T22:07:43Z

How to run this.

check out branch
export a registry variable to something you have push access to export REGISTRY=docker.io/faiq
run make docker-build
run make docker-push
run make release-manifests
apply the release-manfiests in my case it was running

$ k apply -f _artifacts/image-patch/infrastructure-components/source-manifest.yaml

run kubectl set image deployment capa-controller-manager cluster-api-aws-controller-amd64=$REGISTRY/cluster-api-aws-controller-amd64:dev -n capa-system

apply some manifests

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: default-control-plane
  infrastructureRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: default-control-plane
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: default-control-plane
spec:
  addons:
  - name: kube-proxy
    version: v1.32.0-eksbuild.2
  network:
    cni:
      cniIngressRules:
      - description: kube-proxy metrics
        fromPort: 10249
        protocol: tcp
        toPort: 10249
      - description: NVIDIA Data Center GPU Manager metrics
        fromPort: 9400
        protocol: tcp
        toPort: 9400
      - description: Prometheus node exporter metrics
        fromPort: 9100
        protocol: tcp
        toPort: 9100
  region: us-west-2
  sshKeyName: ""
  version: v1.33.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: default2
spec:
  template:
    spec:
      cloudInit:
        insecureSkipSecretsManager: true
      ami:
        eksLookupType: AmazonLinux2023
      instanceMetadataOptions:
        httpTokens: required
        httpPutResponseHopLimit: 2
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
      instanceType: m5a.16xlarge
      rootVolume:
        size: 80
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 # needs to match this thing
kind: NodeadmConfigTemplate
metadata:
  name: default
spec:
  template:
    spec:
      files:
        - path: /etc/yum.repos.d/centos-9.repo
          owner: root:root
          permissions: "0755"
          content: |-
            [baseos]
            name=CentOS Stream 9 - BaseOS
            baseurl=https://mirror.stream.centos.org/9-stream/BaseOS/$basearch/os/
            gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-Official
            gpgcheck=1
            repo_gpgcheck=0
            metadata_expire=6h
            countme=1
            enabled=1

            [appstream]
            name=CentOS Stream 9 - AppStream
            baseurl=https://mirror.stream.centos.org/9-stream/AppStream/$basearch/os/
            gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-Official
            gpgcheck=1
            repo_gpgcheck=0
            metadata_expire=6h
            countme=1
            enabled=1
      kubelet:
        config:
          evictionHard:
            memory.available: "2000Mi"
      preBootstrapCommands:
        - yum -y update; yum -y install iscsi-initiator-utils nfs-utils lvm2 xfsprogs ipvsadm sysstat lsscsi
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: default
spec:
  clusterName: default
  replicas: 3
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: NodeadmConfigTemplate
          name: default
      clusterName: test
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: default2
      version: v1.33.0

dkoshkin

Looking good, just a few questions

config/rbac/role.yaml

bootstrap/eks/api/v1beta2/nodeadmconfig_types.go

bootstrap/eks/controllers/eksconfig_controller.go

bootstrap/eks/internal/userdata/nodeadm.go

bootstrap/eks/controllers/nodeadmconfig_controller.go

bootstrap/eks/api/v1beta2/nodeadmconfig_types.go

bootstrap/eks/controllers/nodeadmconfig_controller.go

bootstrap/eks/internal/userdata/nodeadm.go

bootstrap/eks/api/v1beta2/nodeadmconfig_types.go

supershal

Thanks for implementing and driving this. 🙌

CharlieR-o-o-t · 2025-10-05T21:11:09Z

@faiq , hello! Are you planning to merge this to upstream?

CharlieR-o-o-t · 2025-10-06T10:31:06Z

bootstrap/eks/internal/userdata/nodeadm.go

+  {{- end }}
+  {{- end }}
+{{- if .KubeletConfig }}
+  kubelet:


will result in incorrect NodeConfig generation if only kubelet flags have been provided

Have you tried this out? I can give it a try as well.

Yes, it generates "flags" section under "cluster", but should be inside "kubelet".

"kubelet:" will be rendered only if config privided.

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 kind: NodeadmConfig metadata: annotations: cluster.x-k8s.io/cloned-from-groupkind: NodeadmConfigTemplate.bootstrap.cluster.x-k8s.io cluster.x-k8s.io/cloned-from-name: my-eks-cluster3-md-0-lxc6k cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1" cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1" creationTimestamp: "2025-10-06T18:16:18Z" generation: 1 labels: cluster.x-k8s.io/cluster-name: my-eks-cluster3 cluster.x-k8s.io/deployment-name: my-eks-cluster3-md-0-ph4nh cluster.x-k8s.io/set-name: my-eks-cluster3-md-0-ph4nh-fbpgq machine-template-hash: 1201924711-fbpgq topology.cluster.x-k8s.io/deployment-name: md-0 topology.cluster.x-k8s.io/owned: "" name: my-eks-cluster3-md-0-ph4nh-fbpgq-dgzhn namespace: default ownerReferences: - apiVersion: cluster.x-k8s.io/v1beta1 blockOwnerDeletion: true controller: true kind: Machine name: my-eks-cluster3-md-0-ph4nh-fbpgq-dgzhn uid: c9a7a946-21ff-43ee-bf77-d3bf7f925094 resourceVersion: "24606" uid: 59ad882d-9f02-4288-8d03-88bdf7dd0391 spec: kubelet: flags: - --register-with-taints=key=value:NoExecute

this works fine

CharlieR-o-o-t · 2025-10-06T13:15:25Z

@faiq , if aws secret manager is used (which is true by default), userdata will not be updated in case bootstrap change to NodeadmConfigTemplate. Need to update secret manager userdata also.

faiq force-pushed the faiq/adds-new-nodeadm-bootstrap-type branch from cb41c5b to 51ce7b1 Compare September 22, 2025 21:53

faiq closed this Sep 23, 2025

faiq reopened this Sep 23, 2025

faiq added the ok-to-test label Sep 23, 2025

faiq added 15 commits September 23, 2025 09:21

ci: adds unit tests

c38276b

Merge pull request #2 from nutanix-cloud-native/faiq/unit-tests-on-pr

641b157

ci: adds unit tests

feat: starts bootstrapping new type with kubebuilder

97049ce

feat: adds fields to nodeadm config

7e42d77

refactor: use a common file resolver so nodeadmconfig can use it

7e108ee

feat: implements nodeadm config controller

af310a2

fix: api type issues from copy paste mistakes

3961c8b

test: adds reconciler tests for nodeadm

955a172

fix: runs golangci-lint

c51c989

fix: boundries in userdata

2f2e34d

fix: kubelet logic

66a5d18

fix: adds nodeadmconfigtemplate crd

59fbdbc

fix: nodeadm kubelet config

4e1fa66

fix: handle runtime.Rawextension properly

a961548

e2e: adds test for nodeadm

3b4537a

faiq force-pushed the faiq/adds-new-nodeadm-bootstrap-type branch from 51ce7b1 to 3b4537a Compare September 23, 2025 16:38

faiq self-assigned this Sep 23, 2025

faiq commented Sep 23, 2025

View reviewed changes

bootstrap/eks/api/v1beta2/eksconfig_types.go Show resolved Hide resolved

dkoshkin reviewed Sep 23, 2025

View reviewed changes

supershal reviewed Sep 24, 2025

View reviewed changes

bootstrap/eks/api/v1beta2/nodeadmconfig_types.go Show resolved Hide resolved

supershal reviewed Sep 24, 2025

View reviewed changes

bootstrap/eks/controllers/nodeadmconfig_controller.go Outdated Show resolved Hide resolved

chore: fixes up tests to no longer require TEST_ENV variable

4f4db0c

supershal reviewed Sep 26, 2025

View reviewed changes

bootstrap/eks/internal/userdata/nodeadm.go Show resolved Hide resolved

supershal reviewed Sep 26, 2025

View reviewed changes

bootstrap/eks/api/v1beta2/nodeadmconfig_types.go Outdated Show resolved Hide resolved

fix: we don't need to set eks specific labels

f6d79e2

fix: removes local store options which isn't supported by capa

3537aa5

faiq force-pushed the faiq/adds-new-nodeadm-bootstrap-type branch from b37b672 to 3537aa5 Compare September 26, 2025 18:44

supershal approved these changes Sep 26, 2025

View reviewed changes

fix: use prenodeadm commands name

c434fec

faiq force-pushed the faiq/adds-new-nodeadm-bootstrap-type branch from 76a95a8 to c434fec Compare September 26, 2025 19:05

faiq force-pushed the main branch from 641b157 to 3fe998e Compare September 26, 2025 19:19

faiq merged commit efb53df into main Sep 26, 2025
5 checks passed

faiq mentioned this pull request Oct 2, 2025

🐛 fix: instance error #6

Merged

5 tasks

CharlieR-o-o-t reviewed Oct 6, 2025

View reviewed changes

🌱 feat: adds new nodeadm bootstrap type #1

🌱 feat: adds new nodeadm bootstrap type #1

Uh oh!

Conversation

faiq commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

faiq commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkoshkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

supershal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CharlieR-o-o-t commented Oct 5, 2025

Uh oh!

CharlieR-o-o-t Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

faiq Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

CharlieR-o-o-t Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

faiq Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

CharlieR-o-o-t commented Oct 6, 2025

Uh oh!

Uh oh!

faiq commented Sep 22, 2025 •

edited

Loading

faiq commented Sep 23, 2025 •

edited

Loading

CharlieR-o-o-t Oct 6, 2025 •

edited

Loading