Cilium ENI mode for CAPA

[T-Kukawka]

Same as for Vintage, we will have to make an option for clusters to run in Cilium ENI mode. Our CAPA EKS solution has this already implemented by default

• CAPA managed clusters
• EKS CAPA managed clusters

UPDATE:
During the implementation task, it was discovered that CAPA only supports single CIDR, whereas we use a special CIDR for Cilium in Vintage as well as EKS (which has support for setting: AWSManagedControlPlane.spec.secondaryCidrBlock).

Not to waste our efforts, we will first release the Cilium ENI mode for CAPA marked as prototype. Thanks to this we will have already all the values and schema in place, where the implementation will follow.

Implementation will include support for AWSManagedControlPlane.spec.secondaryCidrBlock same as EKS, which also will involve adding handling of respective security groups and routing tables for added resources.

For the Vintage <-> CAPA migration we have to pay attention such that we do not create the seceondary CIDR once again, as it should be available and migrated from Vintage cluster.

Tasks

Preview Give feedback

1. Cilium ENI mode for CAPA - single CIDR prototype #3183
   1 of 1
   goal/capa-internal-ga team/phoenix +2
   
2. In cilium-app/helm/cilium/templates/cilium-cni-configmap.yaml, set tag selectors for pod security group (or remove if not applicable; there's an open TODO in there)
3. https://github.com/giantswarm/giantswarm/issues/29343

[T-Kukawka]

also blocked by: #2940

[AndiDog]

Blocked by upstream issues in ENI mode #3006

[AndiDog]

We want to try this again with the backport from #3005

[AndiDog]

v1.14.5 seems to work. I created a CAPA WC and pods on worker nodes had connectivity to the internet. Also on newly-added worker nodes.

[calvix]

Let's also not forget about implementing proper pod limits for nodes to account for ENI limits - like we have in vintage - https://github.com/giantswarm/k8scloudconfig/blob/master/files/conf/setup-kubelet-environment

in EKS this is done by AWS, for CAPA we need to configure it

[AndiDog]

kubernetes-sigs/cluster-api-provider-aws#4898 implements the secondary VPC IPv4 CIDR in upstream CAPA. I tried the backported change and it worked fine in our release-2.3 branch.

Next, I need to look into assigning a security group to pods in the secondary CIDR, as we have it in Vintage AWS. And the aforementioned pod limit still needs to be implemented as well.

[T-Kukawka]

we can start with this now: https://github.com/giantswarm/giantswarm/issues/29343

[AndiDog]

From what I can see and remember, this is all done and all CAPA changes were fully upstreamed.