Cannot create RMQ instance on Openshift

[harshac]

Creating a RabbitMQ cluster on Openshift fails with following error:

FailedCreate statefulset/mcs-services-rabbitmq-server create Pod mcs-services-rabbitmq-server-0 in StatefulSet mcs-services-rabbitmq-server failed error: failed to create PVC persistence-mcs-services-rabbitmq-server-0: persistentvolumeclaims "persistence-mcs-services-rabbitmq-server-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

To Reproduce

Steps to reproduce the behavior:

1. Download the release files and extracted them to our local workspace
2. Follow the steps to install the operator using Kubectl (not Helm) from here
3. Allow the anyuid SCC which allows users to run with any UID and any GID
   oc adm policy add-scc-to-user anyuid -z p-rmq-operator -n pivotal-rabbitmq-system
4. Confirm Operator should be installed successfully
5. Confirm CR is available following instructions from here
6. Add the following lines to the ClusterRole named p-rmq-operator-role of the rbac.yaml and re-apply the rbac.yaml

- apiGroups:
  - rabbitmq.pivotal.io
  resources:
  - rabbitmqclusters/finalizers
  verbs:
  - create
 - get
  - patch
  - update

6. Create a RMQ cluster using

apiVersion: rabbitmq.pivotal.io/v1beta1
kind: RabbitmqCluster
metadata:
  name: rmq-cluster
  namespace: rmq
spec:
  image: <Image>
  replicas: 3
  service:
    type: ClusterIP

7. You will see the following errors

 
Warning   FailedCreate   statefulset/mcs-services-rabbitmq-server   create Claim persistence-mcs-services-rabbitmq-server-0 for Pod mcs-services-rabbitmq-server-0 in StatefulSet mcs-services-rabbitmq-server failed error: persistentvolumeclaims "persistence-mcs-services-rabbitmq-server-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

Expected behavior
A rabbitmqCluster should be successfully created on OpenShift

Version information

• RabbitMQ Cluster Operator: [0.7.0-build.41]

[mkuratczyk]

Looking at some other issues (eg. operator-framework/operator-sdk#1736 (comment)) I think create is not enough. Please add update or just try * and if it works. we can see if we can limit the scope if necessary.

[kellymenzel]

This might be the issue I was emailing a couple folks at VMware about. I have been on vacation so I haven't been able to provide an update, but we were able to work around the issue and get a RabbitMQ instance deployed on OpenShift by editing the system:controller:statefulset-controller cluster role. I couldn't figure out the specific permission that resolves the issue, but I added get, create, patch and update verbs on all finalizers and then it worked...

kubectl edit clusterrole system:controller:statefulset-controller

Then add this

- apiGroups:
  - '*'
  resources:
  - '*/finalizers'
  verbs:
  - get
  - create
  - patch
  - update

Hopefully this is just a workaround until better support can be provided for the OpenShift platform but maybe this gives a clue about where the problem lies.

[harshac]

Thanks @kellymenzel. This helped. @j4mcs and I played around with it a bit. We modified the system:controller:statefulset-controller cluster role and narrowed down the permissions as below.

kubectl edit clusterrole system:controller:statefulset-controller

- apiGroups:
  - rabbitmq.com
  resources:
  - rabbitmqclusters/finalizers
  verbs:
  - update

[harshac]

Created a branch with documentation for Openshift - https://github.com/rabbitmq/rabbitmq-website/tree/k8s-openshift

TODO:

1. Verify the steps against an openshift environment
2. Verify necessary verbs for finalizers in the rule for rabbitmq-cluster-operator-role (Step 3 for openshift)
3. Create a PR

[ansd]

The manual cluster role changes described in the proposed documentation are not needed if we merge #303.

[timbrd]

After applying all the necessary cluster role changes for the operator, the statefulset of the rabbitmq cluster was not able to start due to missing permissions:

create Pod rabbitmq-rabbitmq-server-0 in StatefulSet rabbitmq-rabbitmq-server failed error: pods "rabbitmq-rabbitmq-server-0" is forbidden: unable to validate against any security context constraint: [fsGroup: Invalid value: []int64{999}: 999 is not an allowed group spec.initContainers[0].securityContext.securityContext.runAsUser: Invalid value: 999: must be in the ranges: [1000680000, 1000689999] spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 999: must be in the ranges: [1000680000, 1000689999]]

I had to add the service account, which is used by the statefulset, to anyuid scc:

$ oc adm policy add-scc-to-user anyuid system:serviceaccount:<namespace>:rabbitmq-rabbitmq-server