Support a kubelet on the master node for pluggable CNI (calico, canal, etc)

[svrc]

What this PR does / why we need it:
This PR demonstrates deploying kubelets on the master nodes. This is the foundation for using Kubernetes itself to install system-level drivers and addons such as pluggable CNI drivers. This PR includes pluggable CNI drivers for Calico, Canal, and Flannel.

How can this PR be verified?
Kubo-ci tests are forthcoming.

Just master kubelets:

bosh -d cfcr deploy ./cfcr.yml \
 -o ops-files/add-hostname-to-master-certificate.yml \
 -v api-hostname=<your api hostname> \
 -o ops-files/change-cidrs.yml \
 -o ops-files/master-kubelet.yml \
 -v kubedns_service_ip=10.100.200.2 \
-v service_cluster_cidr=10.100.200.0/24 \
-v pod_network_cidr=10.200.0.0/16 \ 
-v first_ip_of_service_cluster_cidr=10.100.200.1

with flannel running as a daemonset:

bosh -d cfcr deploy ./cfcr.yml \
 -o ops-files/misc/scale-to-one-az.yml
 -o ops-files/add-hostname-to-master-certificate.yml \
 -o ops-files/change-cidrs.yml \
-o ops-files/master-kubelet.yml \
-o ops-files/disable-flannel-enable-ipam.yaml \
-o ops-files/allow-privileged-containers.yml \
-o ops-files/use-vm-extensions.yml  \
-o ops-files/cni/kube-flannel.yml  \ #### CNI HERE
-v api-hostname=<your api hostname> \
 -v deployment_name=cfcr \
 -v kubedns_service_ip=10.100.200.2 \
 -v service_cluster_cidr=10.100.200.0/24 \
 -v pod_network_cidr=10.200.0.0/16 \
 -v first_ip_of_service_cluster_cidr=10.100.200.1 

with flannel+calico running as a daemonset

bosh -d cfcr deploy ./cfcr.yml \
 -o ops-files/misc/scale-to-one-az.yml
 -o ops-files/add-hostname-to-master-certificate.yml \
 -o ops-files/change-cidrs.yml \
-o ops-files/master-kubelet.yml \
-o ops-files/disable-flannel-enable-ipam.yaml \
-o ops-files/allow-privileged-containers.yml \
-o ops-files/use-vm-extensions.yml  \
-o ops-files/cni/canal.yml  \ #### CNI HERE
-v api-hostname=<your api hostname> \
 -v deployment_name=cfcr \
 -v kubedns_service_ip=10.100.200.2 \
 -v service_cluster_cidr=10.100.200.0/24 \
 -v pod_network_cidr=10.200.0.0/16 \
 -v first_ip_of_service_cluster_cidr=10.100.200.1 

with calico CNI running as a daemonset (requires intra-cluster L3, should work on GCP, vSphere, Openstack, AWS, not Azure without tweaking manifest to the new vxlan support):

bosh -d cfcr deploy ./cfcr.yml \
 -o ops-files/misc/scale-to-one-az.yml
 -o ops-files/add-hostname-to-master-certificate.yml \
 -o ops-files/change-cidrs.yml \
-o ops-files/master-kubelet.yml \
-o ops-files/disable-flannel-enable-ipam.yaml \
-o ops-files/allow-privileged-containers.yml \
-o ops-files/use-vm-extensions.yml  \
-o ops-files/cni/calico.yml  \ #### CNI HERE
-v api-hostname=<your api hostname>  \
 -v deployment_name=cfcr \
 -v kubedns_service_ip=10.100.200.2 \
 -v service_cluster_cidr=10.100.200.0/24 \
 -v pod_network_cidr=10.200.0.0/16 \
 -v first_ip_of_service_cluster_cidr=10.100.200.1 

Is there any change in kubo-release?
Yes, to enable taints/labels on the master node: cloudfoundry-incubator/kubo-release#333

Is there any change in kubo-ci?
Forthcoming integration/conformance tests with this variant of configuration (and a few pluggable CNIs)

Does this affect upgrade, or is there any migration required?
For master kubelet ops-file, this only adds an extra worker nodes (the masters themselves) to the cluster.

If removing BOSH flannel and adding a pluggable CNI, this requires a BOSH VM recreate and may lead to some cluster network partitions depending on which CNI driver you are switching to.

Swapping pluggable CNIs probably, requires kubectl delete-ing the CNI daemonset, and bosh deploy --recreateing the VMs to clean up any ip link cruft when swapping the CNI ops-file.

Which issue(s) this PR fixes:
N/A

Release note:

Introduce experimental kubelet jobs on the master node with `ops-files/master-kubelets.yml`.  Requires `ops-files/change-cidrs.yml`.
Introduces experimental CNI plugins with one of `ops-files/cni/{kube-flannel,calico,canal}.yml`.  Requires `ops-files/master-kubelets.yml` and `ops-files/disable-flannel-enable-ipam.yaml`.

[cfdreddbot]

✅ Hey svrc-pivotal! The commit authors and yourself have already signed the CLA.

[svrc-personal]

Apologies for the extra commits above, that was workshop-related activity and I thought I was on another branch! will clean up.

[geofffranks]

Any movement on this? It would be a really nice feature to see.

[ananth07reddy]

Are there any updates on this request?

We would like to use Calico CNI plugin in our cluster but it is not working as expected. It seems CFCR doesn't support customized CNI plugin. We see there is an option to update flannel with Calico but we want to use Flannel for the networking and Calico for the Network policy without changing anything in the CFCR deployment. It would be great to have this feature by default in the CFCR to choose CNI plugin.

Please kindly provide any updates on this feature request.

Thanks :)