Cannot use an existingSecret for persistence with multiple namespaces

[lens0021]

Describe the bug

When singleNamespace: false, It seems workflow-controller and server look for the existingSecret in its own namespace, but they have no permission to get it.

Related helm chart

argo-workflows

Helm chart version

0.40.10

To Reproduce

1. I had a secret which has the username and password of a PostgreSQL instance.
2. I've set up a workflow-controller and a server in argo-workflows namespace with the following values:

   # Some details omitted
   controller:
     workflowNamespaces: ["foo", "bar"]
     persistence:
       nodeStatusOffLoad: true
       archive: true
       archiveTTL: 3d
       postgresql:
         host: argo-workflows-postgresql.argo-workflows.svc.cluster.local
         port: 5432
         database: postgres
         tableName: argo_workflows
         userNameSecret:
           name: my-existing-secret
           key: POSTGRESQL_USERNAME
         passwordSecret:
           name: my-existing-secret
           key: POSTGRESQL_PASSWORD

Expected behavior

The controller should be started.

Screenshots

No response

Additional context

The logs from the controller:

time="2024-07-08T08:55:12.195Z" level=info msg="Persistence configuration enabled"
time="2024-07-08T08:55:12.198Z" level=warning msg="Non-transient error: secrets "my-existing-secret" is forbidden: User "system:serviceaccount:argo-workflows:argo-workflows-workflow-controller" cannot get resource "secrets" in API group "" in the namespace "argo-workflows""
time="2024-07-08T08:55:12.198Z" level=fatal msg="Failed to update config: secrets "my-existing-secret" is forbidden: User "system:serviceaccount:argo-workflows:argo-workflows-workflow-controller" cannot get resource "secrets" in API group "" in the namespace "argo-workflows""
Stream closed EOF for argo-workflows/argo-workflows-workflow-controller-6c98cbdcf5-v9ckp (controller)

It seems {{ template "argo-workflows.controller.fullname" . }} is a clusterRole when singleNamespace: false, so

argo-helm/charts/argo-workflows/templates/controller/workflow-controller-cluster-roles.yaml

Lines 129 to 137 in 186065e

	resourceNames:
	{{- if .Values.controller.persistence.postgresql }}
	- {{ .Values.controller.persistence.postgresql.userNameSecret.name }}
	- {{ .Values.controller.persistence.postgresql.passwordSecret.name }}
	{{- end}}
	{{- if .Values.controller.persistence.mysql }}
	- {{ .Values.controller.persistence.mysql.userNameSecret.name }}
	- {{ .Values.controller.persistence.mysql.passwordSecret.name }}
	{{- end}}

will also be in culster scope.