Operators deployed using the generated k8s resources are forbidden to access the CRD

[loicmathieu]

Operators deployed using the generated k8s resources are forbidden to access the CRD.

Steps to reproduce:

• Build the orchestrator
• Deploy the CRD using the generated CRD resource
• Deploy the orchestrator using the generated kubernetes.yml

Then the operator fail to start with:

2024-04-17 08:55:12,320 ERROR [io.fab.kub.cli.inf.imp.cac.Reflector] (vert.x-eventloop-thread-4) listSyncAndWatch failed for model.kestra.io/v1alpha1/kestraflows, will stop: java.util.concurrent.CompletionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1:443/apis/model.kestra.io/v1alpha1/kestraflows?labelSelector=app.kubernetes.io%2Fmanaged-by%3Dkestra-flow-controller&resourceVersion=0. Message: kestraflows.model.kestra.io is forbidden: User "system:serviceaccount:kestra:kestra-orchestrator" cannot list resource "kestraflows" in API group "model.kestra.io" at the cluster scope. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=model.kestra.io, kind=kestraflows, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kestraflows.model.kestra.io is forbidden: User "system:serviceaccount:kestra:kestra-orchestrator" cannot list resource "kestraflows" in API group "model.kestra.io" at the cluster scope, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).

[loicmathieu]

The issue is caused by a ClusterRole that is bonded using a RoleBinding instead of a ClusterRoleBinding.

What is generated:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kestra-flow-cluster-role
  namespace: kestra
rules:
  - apiGroups:
      - model.kestra.io
    resources:
      - kestraflows
      - kestraflows/status
      - kestraflows/finalizers
    verbs:
      - get
      - list
      - watch
      - patch
      - update
      - create
      - delete
 ---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kestra-flow-crd-validating-role-binding
  namespace: kestra
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: josdk-crd-validating-cluster-role
subjects:
  - kind: ServiceAccount
    name: kestra-orchestrator
    namespace: kestra

What should be generated:

```yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kestra-flow-cluster-role
  namespace: kestra
rules:
  - apiGroups:
      - model.kestra.io
    resources:
      - kestraflows
      - kestraflows/status
      - kestraflows/finalizers
    verbs:
      - get
      - list
      - watch
      - patch
      - update
      - create
      - delete
 ---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kestra-flow-crd-validating-role-binding
  namespace: kestra
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: josdk-crd-validating-cluster-role
subjects:
  - kind: ServiceAccount
    name: kestra-orchestrator
    namespace: kestra

[xstefank]

Hi @loicmathieu, I can't reproduce this. It seems that it will always generate ClusterRoleBinding here - https://github.com/quarkiverse/quarkus-operator-sdk/blob/main/core/deployment/src/main/java/io/quarkiverse/operatorsdk/deployment/AddRoleBindingsDecorator.java#L69.

Can you please confirm that you still can reproduce this error?

[nstuart-idexx]

I am seeing the same error, and have some more details around this that maybe helpful.

The generated Role appears to be corect;

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: customService-cluster-role
  namespace: custom-service-operator
rules:
  - ...

Where things go wrong is still in the role binding. The name of the role being bound to is incorrect;

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-service-cluster-role-binding
  namespace: custom-service-operator
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: custom-service-operator
subjects:
  - kind: ServiceAccount
    name: custom-service-operator
    namespace: custom-service-operator

Note, the roleRef.name value here is incorrect here. It seems to just be using the namespace, instead of the name of the ClusterRole being generated, which appears to be the name of the custom resource. As far as I can tell, the role custom-service-operator doesn't exist anywhere, cluster or otherwise.

[nstuart-idexx]

Poking in the code, it looks like this line is the likely culprit. The method expects this to be the name of the controller, which aligns with the ClusterRole definition. However, it's getting passed the serviceName instead.

[metacosm]

@nstuart-idexx thank you for the details, I've opened PR #959 to address the issue.