unexpected redis-secret-init churn

[bobzoller]

Describe the bug

I'm seeing unexpected churn related to argo-cd-argocd-redis-secret-init (job, role, rolebinding, serviceaccount) each time we run a helmwave diff with the argo-cd helm chart. This was happening in version 7.3.6 and is still happening in 7.3.11. helmwave version 0.36.3.

(I realize this could be a helmwave problem, or this could be exposing a helm problem? I'm a bit of a newb at both unfortunately, and I thought I'd file here first because y'all probably understand what the issue might be even if it's not an actual bug in the argo-cd helm chart itself... apologies in advance.)

I'm using the HA mode with autoscaling example.

relevant output:

...
argocd, argo-cd-argocd-redis-secret-init, Job (batch) has been added:
+ apiVersion: batch/v1
+ kind: Job
+ metadata:
+   annotations:
+     helm.sh/hook: pre-install,pre-upgrade
+     helm.sh/hook-delete-policy: before-hook-creation
+   labels:
+     app.kubernetes.io/component: redis-secret-init
+     app.kubernetes.io/instance: argo-cd
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/name: argocd-redis-secret-init
+     app.kubernetes.io/part-of: argocd
+     app.kubernetes.io/version: v2.11.7
+     helm.sh/chart: argo-cd-7.3.11
+   name: argo-cd-argocd-redis-secret-init
+   namespace: argocd
+ spec:
+   template:
+     metadata:
+       labels:
+         app.kubernetes.io/component: redis-secret-init
+         app.kubernetes.io/instance: argo-cd
+         app.kubernetes.io/managed-by: Helm
+         app.kubernetes.io/name: argocd-redis-secret-init
+         app.kubernetes.io/part-of: argocd
+         app.kubernetes.io/version: v2.11.7
+         helm.sh/chart: argo-cd-7.3.11
+     spec:
+       containers:
+       - command:
+         - argocd
+         - admin
+         - redis-initial-password
+         image: quay.io/argoproj/argocd:v2.11.7
+         imagePullPolicy: IfNotPresent
+         name: secret-init
+         resources: {}
+         securityContext:
+           allowPrivilegeEscalation: false
+           capabilities:
+             drop:
+             - ALL
+           readOnlyRootFilesystem: true
+           runAsNonRoot: true
+           seccompProfile:
+             type: RuntimeDefault
+       restartPolicy: OnFailure
+       serviceAccountName: argo-cd-argocd-redis-secret-init

argocd, argo-cd-argocd-redis-secret-init, Role (rbac.authorization.k8s.io) has been added:
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: Role
+ metadata:
+   annotations:
+     helm.sh/hook: pre-install,pre-upgrade
+     helm.sh/hook-delete-policy: before-hook-creation
+   labels:
+     app.kubernetes.io/component: redis-secret-init
+     app.kubernetes.io/instance: argo-cd
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/name: argocd-redis-secret-init
+     app.kubernetes.io/part-of: argocd
+     app.kubernetes.io/version: v2.11.7
+     helm.sh/chart: argo-cd-7.3.11
+   name: argo-cd-argocd-redis-secret-init
+   namespace: argocd
+ rules:
+ - apiGroups:
+   - ""
+   resourceNames:
+   - argocd-redis
+   resources:
+   - secrets
+   verbs:
+   - get
+ - apiGroups:
+   - ""
+   resources:
+   - secrets
+   verbs:
+   - create

argocd, argo-cd-argocd-redis-secret-init, RoleBinding (rbac.authorization.k8s.io) has been added:
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: RoleBinding
+ metadata:
+   annotations:
+     helm.sh/hook: pre-install,pre-upgrade
+     helm.sh/hook-delete-policy: before-hook-creation
+   labels:
+     app.kubernetes.io/component: redis-secret-init
+     app.kubernetes.io/instance: argo-cd
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/name: argocd-redis-secret-init
+     app.kubernetes.io/part-of: argocd
+     app.kubernetes.io/version: v2.11.7
+     helm.sh/chart: argo-cd-7.3.11
+   name: argo-cd-argocd-redis-secret-init
+   namespace: argocd
+ roleRef:
+   apiGroup: rbac.authorization.k8s.io
+   kind: Role
+   name: argo-cd-argocd-redis-secret-init
+ subjects:
+ - kind: ServiceAccount
+   name: argo-cd-argocd-redis-secret-init

argocd, argo-cd-argocd-redis-secret-init, ServiceAccount (v1) has been added:
+ apiVersion: v1
+ automountServiceAccountToken: true
+ kind: ServiceAccount
+ metadata:
+   annotations:
+     helm.sh/hook: pre-install,pre-upgrade
+     helm.sh/hook-delete-policy: before-hook-creation
+   labels:
+     app.kubernetes.io/component: redis-secret-init
+     app.kubernetes.io/instance: argo-cd
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/name: argocd-redis-secret-init
+     app.kubernetes.io/part-of: argocd
+     app.kubernetes.io/version: v2.11.7
+     helm.sh/chart: argo-cd-7.3.11
+   name: argo-cd-argocd-redis-secret-init
+   namespace: argocd

Related helm chart

argo-cd

Helm chart version

7.3.11

To Reproduce

• add the argo-cd chart to a helmwave setup
• create a values.yaml that mirrors the HA mode with autoscaling example
• helmwave apply
• helmwave diff

Expected behavior

no diff

Screenshots

No response

Additional context

No response

[GlacierWalrus]

tl;dr; I think if you're having this issue it's not a problem with the argo helm chart

Long version:
Yesterday I had similar churn on this pod while using the community chart 7.4.3 .

I didn't realise it at the time but I had another cluster issue which was hiding the logs of the redis secret init pod, which might have been complicating things further.

Before I fixed my cluster (which I did with a combination of a minor update and rotating all the worker nodes), I made this work by using the argocli to generate the secret before installing the chart, and installing the chart with redisSecretInit.enabled: false which seemed to make everything work. My hypothesis is that there was some issue creating that secret due to my cluster being broken, but pre-configuring it meant the helm chart could continue installing argo.

Since I fixed my cluster I'm not able to reproduce this issue, even after purging all argo resouces, the secret init container seems to work fine.

I think the secret init churn is a sign of other problems, rather than it being the cause, but like I said I didn't have logs at the time, and I can't reproduce the issue now that I have logs.

I expect the above will be a suitable workaround for anyone who has this issues, but I would once again caution that this seems to by a symptom of a problem, rather than the problem itself. For anyone who doesn't want to use the argo cli, I expect you can just kubectl apply the following yaml

apiVersion: v1
data:
  auth: UjlpdVcyYktYaEFDdTcyUw==  # <- replace this
kind: Secret
metadata:
  name: argocd-redis
  namespace: argocd
type: Opaque

[mmarisetty]

I have tried to use helm chart version 7.5.2 with the below still the same issue persists:
redisSecretInit.enabled: false

#2928 (comment)

[mmarisetty]

redisSecretInit:
enabled: false

With this it will work. Thanks !