Opt-Out AWS VPC CNI (And Any Other EKS "Magics")

[TigerC10]

I wasn't sure if this was an appropriate place to make this feature request since this request has to do with the EKS service rather than the AMI. If I am submitting to the incorrect place, please let me know where to submit this feature request instead.

What would you like to be added:
When an EKS cluster is created, the AWS VPC CNI Plugin is automatically installed to the cluster. I would like a flag or an option as a part of the create cluster to opt-out of having that plugin (and any other "magic" additions) automatically installed.

Why is this needed:
In order to install a different CNI plugin (calico, weave, flannel, etc.), the only way to achieve this well is to build a custom AMI with the desired CNI plugin prefixed with a 00 instead of the standard 10 so that it circumvents the loading of the AWS VPC CNI plugin. If the AWS VPC CNI plugin is never installed, then we can continue to use the standard AMI more easily.

See Also:
aws/amazon-vpc-cni-k8s#214
aws/amazon-vpc-cni-k8s#176

[micahhausler]

The AWS CNI is configured on cluster create and is not managed by the nodes. If you wanted to use a different CNI plugin, you could remove the aws-node daemonset before you join your nodes, so it never gets installed:

kubectl delete --namespace kube-system daemonset/aws-node

We only add core components to get a cluster up and running (CNI, kube-proxy, kube-dns/coreDNS), and currently don't support selection of these on cluster create. If you'd like more options in this, can you create a feature request on https://github.com/aws/containers-roadmap?

Since this isn't a node-level feature, I'm going to close this issue out.

[alfredkrohmer]

@micahhausler This is not entirely true. To remove the AWS VPC CNI plugin, you need to remove the daemonset/aws-node (or add corresponding taints to your nodes and tolerations to the daemonset so that it only runs on instances that you want it to run on) and you need to start kubelet without --network-plugin=cni) – otherwise kubelet will refuse to start because the configured CNI plugin cannot be brought up (because the aws-node container is not running).

The second part – starting kubelet without --network-plugin=cni – needs to be configured on the nodes and an option in the bootstrap.sh would be really useful; we currently do that via sed in the kubelet.service on instance boot for some instances.

[alfredkrohmer]

I actually managed to get this to work with the existing bootstrap.sh by passing --kubelet-extra-args "--network-plugin=".

[gaurav-dalvi]

We tried this option bootstrap.sh by passing --kubelet-extra-args "--network-plugin=". and that did not work. Any workaround on this one ?

[alfredkrohmer]

You need to do this and:

• remove the aws-node daemonset or
• add taints to the subset of your nodes that you start with --kubelet-extra-args "--network-plugin=" and change the corresponding tolerations of the aws-node daemonset

We are doing the latter. We taint the nodes that we start with --kubelet-extra-args "--network-plugin=" with cni=disabled:NoSchedule and have a node anti affinity configured in the aws-node daemonset so that it does not run on nodes tainted with cni=disabled:NoSchedule (but can be tainted with anything else).

[okgolove]

@devkid which network plugin do you use instead of AWS CNI and how to setup it in this configuration?

[alfredkrohmer]

I'm not exactly sure, I would guess setting --network-plugin= disables the CNI mechanism completely?

[okgolove]

As far I understand yes. In this case, what is the point to do it? I mean, your cluster is unusable, isn't it?

[alfredkrohmer]

We only do this on nodes which will only run pods with hostNetwork: true (for performance and other reasons).

[okgolove]

Got it. Thank you.

[lgg42]

@TigerC10 have you built such image with 00-custom-cni-plugin? I'm about to to the same thing for the same reason you said, but I'd rather prefer not to reinvent the wheel and contribute to an existing project.

[ahrkrak]

@TigerC10 @lgg42 See also: https://medium.com/@jeremy.i.cowan/running-calico-on-eks-f3e52ea41271