OCI Cloud Controller Manager (CCM)

oci-cloud-controller-manager is a Kubernetes Cloud Controller Manager implementation (or out-of-tree cloud-provider) for Oracle Cloud Infrastucture (OCI).

Introduction

External cloud providers were introduced as an Alpha feature in Kubernetes 1.6 with the addition of the Cloud Controller Manager binary. External cloud providers are Kubernetes (master) controllers that implement the cloud-provider specific control loops required for Kubernetes to function.

This functionality is implemented in-tree in the kube-controller-manger binary for existing cloud-providers (e.g. AWS, GCE, etc.), however, in-tree cloud-providers have entered maintenance mode and no additional providers will be accepted. Furthermore, there is an ongoing effort to remove all existing cloud-provider specific code out of the Kubernetes codebase.

Compatibility matrix

	Min Kubernetes Version	Max Kubernetes Version
>=v 0.11	v1.16	v1.18
>=v 0.12	v1.18	v1.21
>=v 0.13	v1.19	v1.21
v1.19.12	v1.19	v1.21
v1.22.0	v1.22	-
v1.23.0	v1.23	-
v1.24.2	v1.24	-
v1.25.2	v1.25	-
v1.26.4	v1.26	-
v1.27.3	v1.27	-
v1.28.2	v1.28	-
v1.29.1	v1.29	-
v1.30.0	v1.30	-

Note: Versions older than v1.28.2 are no longer supported, new features / bug fixes will be available in v1.28.2 and later.

Implementation

Currently oci-cloud-controller-manager implements:

• NodeController - updates nodes with cloud provider specific labels and addresses, also deletes kubernetes nodes when deleted from the cloud-provider.
• ServiceController - responsible for creating load balancers when a service of type: LoadBalancer is created in Kubernetes.

Additionally, this project implements a container-storage-interface, a flexvolume driver and a flexvolume provisioner for Kubernetes clusters running on Oracle Cloud Infrastructure (OCI).

Setup and Installation

To get the CCM running in your Kubernetes cluster you will need to do the following:

1. Prepare your Kubernetes cluster for running an external cloud provider.
2. Create a Kubernetes secret containing the configuration for the CCM.
3. Deploy the CCM as a DaemonSet.

Note: For the setup and installation of flexvolume driver, flexvolume provisioner and container-storage-interface please refer linked resources.

Preparing Your Cluster

To deploy the Cloud Controller Manager (CCM) your cluster must be configured to use an external cloud-provider.

This involves:

• Setting the --cloud-provider=external flag on the kubelet on all nodes in your cluster.
• Setting the --provider-id=<instanceID> flag on the kubelet on all nodes in your cluster. Where <instanceID> is the instance ocid of a node (unique for each node).
• Setting the --cloud-provider=external flag on the kube-controller-manager in your Kubernetes control plane.

Depending on how kube-proxy is run you may need the following:

• Ensuring that kube-proxy tolerates the uninitialised cloud taint. The following should appear in the kube-proxy pod yaml:

- effect: NoSchedule
  key: node.cloudprovider.kubernetes.io/uninitialized
  value: "true"

If your cluster was created using kubeadm >= v1.7.2 this toleration will already be applied. See kubernetes/kubernetes#49017 for details.

Remember to restart any components that you have reconfigured before continuing.

Authentication and Configuration

An example configuration file can be found here. Download this file and populate it with values specific to your chosen OCI identity and tenancy. Then create the Kubernetes secret with the following command:

For CCM -

$ kubectl  create secret generic oci-cloud-controller-manager \
     -n kube-system                                           \
     --from-file=cloud-provider.yaml=provider-config-example.yaml

Note that you must ensure the secret contains the key cloud-provider.yaml rather than the name of the file on disk.

Deployment

Deploy the controller manager and associated RBAC rules if your cluster is configured to use RBAC (replace ? with the version you want to install to):

$ export RELEASE=?
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-cloud-controller-manager-rbac.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-cloud-controller-manager.yaml

Check the CCM logs to ensure it's running correctly:

$ kubectl -n kube-system get po | grep oci
oci-cloud-controller-manager-ds-k2txq   1/1       Running   0          19s

$ kubectl -n kube-system logs oci-cloud-controller-manager-ds-k2txq
I0905 13:44:51.785964       7 flags.go:52] FLAG: --address="0.0.0.0"
I0905 13:44:51.786063       7 flags.go:52] FLAG: --allocate-node-cidrs="false"
I0905 13:44:51.786074       7 flags.go:52] FLAG: --alsologtostderr="false"
I0905 13:44:51.786078       7 flags.go:52] FLAG: --cloud-config="/etc/oci/cloud-config.cfg"
I0905 13:44:51.786083       7 flags.go:52] FLAG: --cloud-provider="oci"

Upgrade

The following example shows how to upgrade the CCM, FVP, FVD and CSI from an older version (replace ? with the version you're upgrading to):

$ export RELEASE=?
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-cloud-controller-manager-rbac.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-cloud-controller-manager.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-volume-provisioner.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-volume-provisioner-rbac.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-flexvolume-driver.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-flexvolume-driver-rbac.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-csi-controller-driver.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-csi-node-driver.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-csi-node-rbac.yaml

Examples

• Service type: LoadBalancer basic NGINX example
• Service type: LoadBalancer NGINX SSL example

Development

See DEVELOPMENT.md.

Support

If you think you've found a bug, please raise an issue.

Contributing

This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide

Security

Please consult the security guide for our responsible security vulnerability disclosure process

License

Copyright (c) 2017, 2023 Oracle and/or its affiliates. All rights reserved.

oci-cloud-controller-manager is licensed under the Apache License 2.0.

See LICENSE for more details.