ArgoCD Image Updater logs says image is updated, but it's not.

[mrksph]

Describe the bug
When using ArgoCD with ArgoCD Image Updater (v0.15.0, also tested with v0.14.0) and ApplicationSet to deploy a demo API to a k3s cluster, the Image Updater successfully updates the image in the ArgoCD application spec, but the pod does not restart to pull the updated image. The updated image is hosted on GitHub Container Registry (GHCR) as a private repository, and credentials are correctly configured.

ArgoCD Image Updater logs shows that it correctly downloads and updates the image (I see the image download count increase in the repo packages page) but says it is pending spec update (dry run=false):

Successfully updated image 'ghcr.io/mrksph/demo-api@dummy' to 'ghcr.io/mrksph/demo-api@sha256:3039e1fc83a2e3e2400b842a03fc8b4669a9f9b3ef16466b242d8b489f21637e', but pending spec update (dry run=false)"

I remember I did manage to set this up some months ago. Now that I'm back with it, I can't make it work.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a demo API using ArgoCD, ArgoCD Image Updater, and ApplicationSet.
2. Configure the private image from GHCR and ensure credentials are set up correctly.
3. Apply the following folder structure:

   |-argocd-bootstrap.yaml
   |-app-of-apps/
   |--applicationset.yaml
   |-dev/
   |--demo-api/
   |---values.yaml
   |---templates/
   |---# All Helm template files... (deployment, ingress, application, service)
   

4. Files:
   argocd-bootstrap.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root-app
namespace: argocd
spec:
project: default
source:
  repoURL: 'https://github.com/mrksph/k3s-manifests'
  targetRevision: HEAD
  path: app-of-apps
  directory:
    recurse: true
destination:
  server: 'https://kubernetes.default.svc'
  namespace: argocd
syncPolicy:
  automated:
    prune: true
    selfHeal: true

applicationset.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: applications
namespace: argocd
spec:
generators:
  - git:
      repoURL: https://github.com/mrksph/k3s-manifests.git
      revision: HEAD
      directories:
        - path: dev/*     # Match all subdirectories in the dev folder
template:
  metadata:
    name: '{{path.basename}}-{{path[1]}}'
    annotations:
      argocd-image-updater.argoproj.io/image-list: "{{path.basename}}=ghcr.io/<username>/{{path.basename}}:{{path[0]}}"
      argocd-image-updater.argoproj.io/{{path.basename}}.update-strategy: "digest"
  spec:
    project: default
    source:
      repoURL: https://github.com/mrksph/k3s-manifests.git
      targetRevision: HEAD
      path: '{{path}}'
    destination:
      server: https://kubernetes.default.svc
      namespace: '{{path.basename}}-{{path[0]}}'
    syncPolicy:
      automated:
        prune: true
        selfHeal: true
      syncOptions:
        - CreateNamespace=true

dev/demo-api/values.yaml

replicaCount: 1

image:
repository: ghcr.io/<username>/<repository>
pullPolicy: IfNotPresent
tag: <tag> # I built my demo-api with "dev" tag, and set it here

imagePullSecrets:
- name: ghcr-secret

service:
type: ClusterIP
port: 8080

ingress:
# Configure your ingress

resources: {}

nodeSelector: {}

tolerations: []

affinity: {}

5. Observe the logs to see the image update confirmation.
6. Check if the pod restarts to pull the updated image.

Expected behavior
After the image is successfully updated in the ArgoCD spec, the associated pod should restart and pull the updated image.

Additional context

• The image repository is private, and credentials have been confirmed as correct.
• Helm charts are used for the deployment templates.
• ArgoCD is configured with ApplicationSet and Image Updater to automate updates.

Version

• ArgoCD Image Updater: v0.15.0 (also tested with v0.14.0)
• ArgoCD: [Version details, if available]
• Kubernetes: [Cluster version, if relevant]

Logs

time="2024-12-06T18:21:29Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2024-12-06T18:21:29Z" level=debug msg="Processing application demo-api-demo-api"
time="2024-12-06T18:21:29Z" level=debug msg="Considering this image for update" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dev registry=ghcr.io
time="2024-12-06T18:21:29Z" level=debug msg="Using version constraint 'dev' when looking for a new tag" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dev registry=ghcr.io
time="2024-12-06T18:21:30Z" level=debug msg="found 1 from 1 tags eligible for consideration" image="ghcr.io/mrksph/demo-api:dev"
time="2024-12-06T18:21:30Z" level=info msg="Setting new image to ghcr.io/mrksph/demo-api@sha256:3039e1fc83a2e3e2400b842a03fc8b4669a9f9b3ef16466b242d8b489f21637e" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dummy registry=ghcr.io
time="2024-12-06T18:21:30Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=demo-api-demo-api image=ghcr.io/mrksph/demo-api
time="2024-12-06T18:21:30Z" level=info msg="Successfully updated image 'ghcr.io/mrksph/demo-api@dummy' to 'ghcr.io/mrksph/demo-api@sha256:3039e1fc83a2e3e2400b842a03fc8b4669a9f9b3ef16466b242d8b489f21637e', but pending spec update (dry run=false)" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dummy registry=ghcr.io
time="2024-12-06T18:21:30Z" level=debug msg="Using commit message: "
time="2024-12-06T18:21:30Z" level=info msg="Committing 1 parameter update(s) for application demo-api-demo-api" application=demo-api-demo-api
time="2024-12-06T18:21:30Z" level=info msg="Successfully updated the live application spec" application=demo-api-demo-api
time="2024-12-06T18:21:30Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"

[mrksph]

Just read this article. The issue probably arises from the fact I'm using ApplicationSet. ArgoCD Image Updater tries to update the Application resource but as Im using ApplicationSet they somehow conflict and the pod doesn't restart / the image doesn't updates.

@chengfang what's the suggested way of handling image updater along with ApplicationSet?

[chengfang]

Related discussion in #400, #431

[nia-potato]

i have the exact same issues and tried the workarounds in the linked article above and the application policy, and still have the same issue, i am using kustomize patches in the appset. Are there any other workarounds for this issue?

[pablo-tx]

Like @nia-potato, I also use kustomize but I don't use an ApplicationSet and I have the same issue.

All the related issues point to ApplicationSet as the culprit but it doesn't seem to be the case.

[chengfang]

@mrksph in the description you mentioned using image-updater v0.15.0. Not sure if you have tried the patch release v0.15.1, which fixed a bug with app-of-apps: #896 .

@pablo-tx for non-applicationset cases, things to double check:

• if argocd auto-sync is enabled to sync from git repo
• if the interval has passed
• if argocd syncs from the repo and branch that contains the updates from image-updater

[nia-potato]

hi @chengfang i have tried with the latest version v0.15.1

and have tried it with a application instead of applicationset with the three things you mentioned to check, with argocd write-back method, but the issue still seems to be persisting. Is there anything else i should check?

[mrksph]

@mrksph in the description you mentioned using image-updater v0.15.0. Not sure if you have tried the patch release v0.15.1, which fixed a bug with app-of-apps: #896 .

@pablo-tx for non-applicationset cases, things to double check:

• if argocd auto-sync is enabled to sync from git repo
• if the interval has passed
• if argocd syncs from the repo and branch that contains the updates from image-updater

sorry for the late reply, I made it work following the article I mentioned.

[pablo-tx]

@chengfang the auto-sync works well, it updates the image properly when there are changes.

My particular problem is that the messages mentioned in the issue are recurrent every 2 minutes even if there isn't any change.

I use argocd as write back method

[nia-potato]

i have the exact same issue with @pablo-tx when testing with git write back, @chengfang any ideas? it basically goes into a loop of spamming the PR/MR.