IBM Cloud Object Storage plug-in

IBM Cloud Object Storage plug-in is a Kubernetes volume plug-in that enables Kubernetes pods to access IBM Cloud Object Storage buckets. The plug-in has two components: a dynamic provisioner and a FlexVolume driver for mounting the buckets using s3fs-fuse on a worker node.

Prerequisites

Before installing IBM Cloud Object Storage plug-in in a Kubernetes cluster, ensure:

1. RBAC should be enabled for the Kubernetes cluster.
   
2. S3FS-FUSE should be installed on every worker node in the cluster.

Build the provisioner image and driver binary

For building the provisioner image and the driver binary, docker, GO and glide should be installed on your local system.

1. On your local machine, install docker, Go, and glide.
2. Set the GOPATH environment variable.
3. Build the provisioner container image and the driver binary
   clone the repo or your forked repo

   $ mkdir -p $GOPATH/src/github.com/IBM
   $ mkdir -p $GOPATH/bin
   $ cd $GOPATH/src/github.com/IBM/
   $ git clone https://github.com/IBM/ibmcloud-object-storage-plugin.git
   $ cd ibmcloud-object-storage-plugin
   

   build project and runs testcases

   $ make
   

   build container image for the provisioner

   $ make provisioner
   

   build driver binary

   $ make driver
   

   You can find the driver binary under $GOPATH/bin directory with name ibmc-s3fs.
   Run docker images command to view the provisioner container image by name ibmcloud-object-storage-plugin.

Provisioner image

Push the provisioner container image from the build system to your image repository, the one being used for your Kubernetes cluster. Refer to docker push

Installation

1. Copy driver binary ibmc-s3fs from your build system to each worker node, under /tmp/

2. On every worker node execute following commands to copy the driver binary ibmc-s3fs to Kubernetes plugin directory:

   $ sudo mkdir -p /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ibm~ibmc-s3fs
   $ sudo cp /tmp/ibmc-s3fs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ibm~ibmc-s3fs
   $ sudo chmod +x /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ibm~ibmc-s3fs/ibmc-s3fs
   $ sudo systemctl restart kubelet
   

3. Create the provisioner.
   Before executing following command update image details as per your repository in deploy/provisioner.yaml. Currently, it is image: ibmcloud-object-storage-plugin:latest, which expects image to be in public docker hub

   $ kubectl create -f deploy/provisioner-sa.yaml
   $ kubectl create -f deploy/provisioner.yaml
   

4. Create the storage class

   $ kubectl create -f deploy/ibmc-s3fs-standard-StorageClass.yaml
   

Verify IBM Cloud Object Storage plug-in installation

$ kubectl get pods -n kube-system | grep object-storage
  ibmcloud-object-storage-plugin-7c96f8b6f7-g7v98   1/1       Running   0          28s

$ kubectl get storageclass |grep s3
  ibmc-s3fs-standard           ibm.io/ibmc-s3fs

Usage

Create Secret

To enable the plug-in to access the object storage, you need to share your access keys as secret. If you want to use IBM Cloud Object Storage access keys, then use access-key and secret-key.
If you want to use IBM IAM OAuth instead of access-key, use api-key and service-instance-id.

All keys should be encoded in base64 using echo -n "<key_value>" | base64

Create secret:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: ibm/ibmc-s3fs
metadata:
  name: test-secret
  namespace: <NAMESPACE_NAME>
data:
  access-key: <access key encoded in base64 (when not using IAM OAuth)>
  secret-key: <secret key encoded in base64 (when not using IAM OAuth)>
  api-key: <api key encoded in base64 (for IAM OAuth)>
  service-instance-id: <service-instance-id encoded in base64 (for IAM OAuth + bucket creation)>
EOF

Note: Replace <NAMESPACE_NAME> with your namespace (for example: default).
The secret and PVC should be created in same namespace.

Create a PVC and POD

1. Create PVC.
   

   kubectl apply -f - <<EOF
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: s3fs-test-pvc
     namespace: <NAMESPACE_NAME>
     annotations:
       volume.beta.kubernetes.io/storage-class: "ibmc-s3fs-standard"
       ibm.io/auto-create-bucket: "true"
       ibm.io/auto-delete-bucket: "false"
       ibm.io/bucket: "<BUCKET_NAME>"
       ibm.io/object-path: ""    # Bucket's sub-directory to be mounted (OPTIONAL)
       ibm.io/endpoint: "https://s3-api.dal-us-geo.objectstorage.service.networklayer.com"
       ibm.io/region: "us-standard"
       ibm.io/secret-name: "test-secret"
       ibm.io/stat-cache-expire-seconds: ""   # stat-cache-expire time in seconds; default is no expire.
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 8Gi # fictitious value
   EOF
   

   Note: Replace <BUCKET_NAME> and <NAMESPACE_NAME>.
   The secret and PVC should be in same namespace.
   For end-point and region refer to AWS CLI.

2. Verify the PVC, s3fs-test-pvc, creation.

   $ kubectl get pvc -n <NAMESPACE_NAME>
   NAME            STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS         AGE
   s3fs-test-pvc   Bound     pvc-9167eace-b194-11e7-bc69-dab1a668f971   8Gi        RWO           ibmc-s3fs-standard   35s
   

   If STATUS is Bound then the PVC has been created successfully.

3. Create a POD using the PVC.
   

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: s3fs-test-pod
     namespace: <NAMESPACE_NAME>
   spec:
     containers:
     - name: s3fs-test-container
       image: anaudiyal/infinite-loop
       volumeMounts:
       - mountPath: "/mnt/s3fs"
         name: s3fs-test-volume
     volumes:
     - name: s3fs-test-volume
       persistentVolumeClaim:
         claimName: s3fs-test-pvc
   EOF
   

4. Verify the POD and the volume.
   Verify that the s3fs-test-pod POD is in Running state.
   

   $ kubectl get pods -n <NAMESPACE_NAME> | grep s3fs-test-pod
     s3fs-test-pod                 1/1       Running   0          28s
   

   Get into the POD kubectl exec -it s3fs-test-pod -n <NAMESPACE_NAME> bash
   and run the following commands to verify access to the mounted bucket:

   $ kubectl exec -it s3fs-test-pod -n <NAMESPACE_NAME> bash
   root@s3fs-test-pod:/#
   root@s3fs-test-pod:/# df -Th | grep s3
   s3fs           fuse.s3fs  256T     0  256T   0% /mnt/s3fs
   
   root@s3fs-test-pod:/# cd /mnt/s3fs/
   root@s3fs-test-pod:/mnt/s3fs# ls
   root@s3fs-test-pod:/mnt/s3fs#
   
   root@s3fs-test-pod:/mnt/s3fs# echo "IBM Cloud Object Storage plug-in" > sample.txt
   root@s3fs-test-pod:/mnt/s3fs# ls
   sample.txt
   root@s3fs-test-pod:/mnt/s3fs# cat sample.txt
   IBM Cloud Object Storage plug-in
   root@s3fs-test-pod:/mnt/s3fs#
   

Use Custom CA Bundle

Note: It is recommended to expose Kube Dns on Worker Nodes before performing below steps.

Pass the ca-bundle key in the cos secret with parameter ca-bundle-crt along with access-key and secret-key.

Sample Secret:

 apiVersion: v1
 kind: Secret
 type: ibm/ibmc-s3fs
 metadata:
   name: test-secret
   namespace: <NAMESPACE_NAME>
 data:
   access-key: <access key encoded in base64 (when not using IAM OAuth)>
   secret-key: <secret key encoded in base64 (when not using IAM OAuth)>
   api-key: <api key encoded in base64 (for IAM OAuth)>
   service-instance-id: <service-instance-id encoded in base64 (for IAM OAuth + bucket creation)>
   ca-bundle-crt: < TLS Public cert bundles encoded in base64>

Create PVC by providing COS-Service name and COS-Service namespace

Sample PVC template:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: s3fs-test-pvc
  namespace: <NAMESPACE_NAME>
  annotations:
    volume.beta.kubernetes.io/storage-class: "ibmc-s3fs-standard"
    ibm.io/auto-create-bucket: "true"
    ibm.io/auto-delete-bucket: "false"
    ibm.io/bucket: "<BUCKET_NAME>"
    ibm.io/object-path: ""    # Bucket's sub-directory to be mounted (OPTIONAL)
    ibm.io/region: "us-standard"
    ibm.io/secret-name: "test-secret"
    ibm.io/stat-cache-expire-seconds: ""   # stat-cache-expire time in seconds; default is no expire.
    ibm.io/cos-service: <COS SERVICE NAME>
    ibm.io/cos-service-ns: <NAMESPACE WHERE COS SERVICE IS CREATED>
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi # fictitious value

Uninstall

Execute the following commands to uninstall/remove IBM Cloud Object Storage plugin from your Kubernetes cluster:

$ kubectl delete deployment ibmcloud-object-storage-plugin -n kube-system
$ kubectl delete clusterRoleBinding ibmcloud-object-storage-plugin ibmcloud-object-storage-secret-reader
$ kubectl delete clusterRole ibmcloud-object-storage-plugin ibmcloud-object-storage-secret-reader
$ kubectl delete sa ibmcloud-object-storage-plugin -n kube-system
$ kubectl delete sc ibmc-s3fs-standard