[EKS] [request]: Cluster init job to bootstrap new clusters

[bryantbiggs]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
This is somewhat related to #923 (reliant on it actually) but I didn't want to muck up that feature request with additional noise since on its own its quite good and would be a great addition.

When provisioning new clusters, there are a number of steps that unfortunately have to be manually performed that are repetitious; especially when treating clusters as temporary/ephemeral (i.e. - blue/green cluster upgrades, sandbox clusters, etc.). An example would be:

1. Provision cluster and a managed node group (at least one node so that the core service pods come up happy and the provisioning succeeds. There is an alternate route where you don't deploy a node group yet, make the necessary changes, and then deploy your node groups. Its 6 vs half a dozen in the end though regarding which route you take)
2. Once the cluster and node group are deployed, the first step we'll look at is pod networking:
   a. One route is to remove the VPC CNI permissions from the node group and rely on an IRSA for providing the VPC CNI the necessary permissions where the VPC CNI is a managed addon. I cannot launch a node group without the VPC CNI permissions and rely only on the IRSA role for the addon because its a chicken vs the egg problem. I have to provision first with the node having the permissions, then come back and remove and rely on the IRSA only
   b. Another route is where I don't want to use the VPC CNI and I need to delete it and provision the desired CNI/configuration. Again, this can only be done manually since EKS makes the default assumption to use the VPC CNI (I assume for a good "out of the box" experience, which makes sense)

• Note: the other managed addons that are provided by default can also fall within a similar pattern. In my experience, folks want to codify what goes into their cluster. Trying to codify something that was provisioned outside of your control is one of the core issues here and why [EKS] [request]: API flag to initialize completely bare EKS cluster #923 is tied directly to this topic

3. Lastly, at this point we want to switch to our preferred cluster management solution which usually introduces another chicken vs. the egg situation. This primarily concerns folks who are following GitOps practices as well as folks who are trying to follow recommended practices by disabling the cluster control plane public endpoint.
   • GitOps is great for clusters that do not have a publicly accessible endpoint; their operators sit within the cluster and pull changes in from external manifest repositories, image registries, etc. (pull method vs push)
   • However, how do we first provision our GitOps operator to start this pull based process - I've seen hacks where clusters are provisioned with public endpoints enabled, push deploy the GitOps oeprator with a script, then remove the public endpoint access. Its a cumbersome process to say the least. This is where having a job that is again sitting within the cluster or at least within the same network (access) as the cluster, we can move that bootstrapping to this job that is executed once when creating a new cluster.
   • Once the GitOps operator is provisioned, all changes can be handled via the codified manifests that the operator monitors for changes

✨ The Request ✨

The ability to provide a cluster init job that is run once upon cluster creation (think user data, but for clusters).

The changes in #923 are required to make this work without race conditions. If default components are installed, its a race to get the default components removed from the cluster before nodes come up using them and the desired components installed (i.e. - CoreDNS, CNI, etc.). If they are not on the cluster to begin with, users are simply providing their desired configurations via the init job. The init job needs to run in a loop for those tools that are not designed as operators; but an operator (which itself is its own control loop) is the preferred approach. The longer form for ArgoCD is shown below - the preferred alternative would be to use the ArgoCD operator instead.

This init job also alleviates the shortcomings of running a Fargate only cluster where users have to manually intervene to configure CoreDNS for Fargate

This lifecycle roughly looks like the following:

stateDiagram-v2
[*] --> ControlPlane
ControlPlane --> InitJob
InitJob --> InitJobRetry
InitJobRetry --> InitJob
ControlPlane --> NodeGroup
ControlPlane --> FargateProfile
InitJob --> [*]
NodeGroup --> [*]
FargateProfile --> [*]

Loading

1. Users provision a bare cluster. When they submit the cluster creation, they have the ability to provide an init job spec; see a rough example below
2. The control plane will be provisioned as usual, VPC endpoints attached into client account, etc.
3. Following creation of the control plane, the init job is kicked off which runs asynchronously to node group/profile creation. Ideally, users should employ an operator pattern for their bootstrapping mechanism of choice which has its own self-contained control loop, but the init job itself can also be configured to backoff and retry x number of times for up to y timeout (standard job specs within the manifest). The key will be how to properly surface failures to users when the init job does not succeed - because nodes will most likely fail to join the cluster as well in that scenario.
4. Upon successful execution of the init job, nodes should join the cluster successfully and the cluster bootstrap process is complete. I am positive that there are other use cases that people employ in bootstrapping a fresh cluster; this is just the most common set of steps that I have encountered when creating clusters or helping clients with their clusters (i.e - what steps are folks taking for restoring persistent volumes or handling stateful sets, etc. when bootstrapping a new cluster)

apiVersion: batch/v1
kind: Job
metadata:
  name: clusterInit
spec:
  template:
    spec:
      containers:
        - name: al2
          # most likely will need a new image which has kubectl installed by default to match cluster version
          image: public.ecr.aws/amazonlinux/amazonlinux:2
          command:
            - /bin/sh
            - -c
            - |
              # we will assume kubectl is installed either via the image or script
              # and the pod will most likely require IRSA
              kubectl create namespace argocd
              kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
              ARGOCD_OPTS='--port-forward-namespace argocd'
              ARGOCD_INIT_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo)
              ARGOCD_NEW_PWD=$(aws secretsmanager get-secret-value --secret-id prod/mysecret)
              argocd account update-password --current-password $ARGOCD_INIT_PWD --new-password $ARGOCD_NEW_PWD
              # Bootstrap cluster using app of apps pattern https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/
              # One suite of apps could be management apps - CoreDNS, Cillium CNI, etc., another for monitoring/observability,
              # andother for applications, etc. Up to the cluster administrator(s) to decide
              argocd app create apps \
                --dest-namespace argocd \
                --dest-server https://kubernetes.default.svc \
                --repo https://github.com/argoproj/argocd-example-apps.git \
                --path apps
              argocd app sync apps \
                --async \ # async is important since it won't succeed until nodes are launched
                --retry-backoff-duration 2m  \ # add some backoff and retries to give time for nodes to launch
                --retry-limit 5 \
                --timeout 630
      restartPolicy: Never

Which service(s) is this request for?
EKS

Are you currently working around this issue?
A mix of hacky scripts and manual intervention - "playbooks" if you will

Additional context

• we need the abilty for setting custom cni networking upon creating a cluster terraform-aws-modules/terraform-aws-eks#1822

• [EKS] [request]: API flag to initialize completely bare EKS cluster #923 (comment)

• [EKS] [CNI]: Optional Default CNI Plugin Installation #71

• [EKS] [request]: 1-click upgrades support blue/green or reversibility #1592

• [EKS] [addon]: Support Flux V2 in EKS addons #1593

• Opt-Out AWS VPC CNI (And Any Other EKS "Magics") awslabs/amazon-eks-ami#117

• Ability to choose another CNI amazon-vpc-cni-k8s#214

• What are consequences of disabling the CNI plugin in EKS? amazon-vpc-cni-k8s#176

• https://github.com/aws-quickstart/quickstart-amazon-eks/issues/51

• [EKS] [request]: Manage IAM identity cluster access with EKS API #185 is tangentially related due to the various hacks that people have to employ to initially provision access into the cluster via a codified process. With the approach here, this can help alleviate some of those issues - instead of giving access to the role that created the cluster (usually a CI/CD role that is quite elevated and heavily protected), users can offload the aws-auth configmap to be managed by their GitOps operator and/or they can update the configmap upon cluster creation

[stevehipwell]

@bryantbiggs I think #1559 would be an alternative to this, assuming the behaviours required were provided for. If I ever get some free time it's near the top of my project list, although I might create a more generic operator suitable for all kubeadm (style) clusters.

For EKS my use case is managing the default kubeadm services (CoreDNS & kube-proxy) and the CNI (aws-vpc-cni). I'd also like to see it optionally support the "required" services (aws-load-balanacer-controller, cluster-autoscaler, aws-node-termination-handler, metrics-server, etc).

[bryantbiggs]

@bryantbiggs I think #1559 would be an alternative to this, assuming the behaviours required were provided for. If I ever get some free time it's near the top of my project list, although I might create a more generic operator suitable for all kubeadm (style) clusters.

For EKS my use case is managing the default kubeadm services (CoreDNS & kube-proxy) and the CNI (aws-vpc-cni). I'd also like to see it optionally support the "required" services (aws-load-balanacer-controller, cluster-autoscaler, aws-node-termination-handler, metrics-server, etc).

Not quite - I am looking to avoid this statement from #1559 Initially this operator could be manually installed into a https://github.com/aws/containers-roadmap/issues/923 and used to manage add-ons.

The overall idea I am pitching here is two parts, mostly centered on the 2nd:

1. Bare cluster free of any defaults - this is covered nicely by [EKS] [request]: API flag to initialize completely bare EKS cluster #923
2. A way to programmatically bootstrap a cluster where (ideally) I can get my gitops operator installed and that operator takes over everything else from there. This is where the idea of a "user data" type flow came from - just light bootstrapping so that cluster configuration is passed off to another tool that is running within the cluster. The "user data" flow is key because I want to codify it and avoid manual intervention between provisioning the EKS resources and my gitops operator provisioning cluster resources

[stevehipwell]

@bryantbiggs I suspect that any solution to this is going to require compute somewhere; this issue would need something to run the bootstrap script which IMHO makes it equivalent to #1559.

I'm pretty sure you could do this already with a Lambda function, for TF a aws_lambda_invocation resource. You could then run further config once this function has completed. I don't think you'd have a dependency on #923 as your bootstrap could clean all of this up before any nodes are connected. This is something that could be added to the OSS EKS TF module; expose a variable to pass in a function to execute on create and maybe a peer module to create the lambda and execute some shell script with the correct dependencies.

For #1559, the operator could be run in Fargate (like Karpenter can be for the same reason) and would be expected to use host networking so can precede CNI configuration. Obviously with an operator it wouldn't matter if it was run on a cluster with incorrect config and attached nodes as it's running a desired state control loop with eventual consistency.

[jwenz723]

Couldn't you use terraform to execute the commands which you want to occur at cluster creation time?

For example:

• Here is an example of how to create an EKS cluster: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/771465be280450fc96d889ef9e15f191bb512849/examples/karpenter/main.tf#L74
• After the EKS cluster has been created, install karpenter into the cluster: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/771465be280450fc96d889ef9e15f191bb512849/examples/karpenter/main.tf#L157
• And here is an example of how to execute an arbitrary kubernetes manifest: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/771465be280450fc96d889ef9e15f191bb512849/examples/karpenter/main.tf#L207

In this example they are creating a fargate profile to host the karpenter app.

[bryantbiggs]

yes, I am familiar with those since I created them 😬 - this request was more about how best to transition from infrastructure provisioning over to cluster provisioning smoothly and seamlessly. The somewhat ideal, high-level flow being:

1. Create cluster
2. Once the control plane is ready, a GitOps controller is installed (this was sort of the crux of the "bootstrap" idea) and configured to point at the appropriate manifest repositories
3. The GitOps controller provisions/reconciles the resources defined at the target manifest location(s)

There are of course other, more complex scenarios such as standing up a cluster using a different CNI without chaining, managing core addons such as CoreDNS through the GitOps controller, etc.

But thats the gist of it - provision the cluster, somehow get a controller installed and pointed at manifests, said controller reconciles the intended cluster state. No hacks, no manual intervention steps, not multi-step applies required (ideally)

[gmolaire]

Any progress on this issue? Or everyone is using the workaround?