Image Updater does not work with "App in any namespace" Argo configuration

[renchap]

Describe the bug
App in any namespace is a set of Argo feature allowing you to have Application resources in other namespaces than argocd.

This setup is not supported by Image Updater

To Reproduce
Steps to reproduce the behavior:

• Configure an Argo app living in another namespace according to the docs above
• Setup Image Updater annotations on it
• Run Image Updater and see that it does not pick the application

Expected behavior
Image Updater detects the Application in other namespaces than argocd

Additional context

As the Application was not detected using the kubernetes application API, I tried switching to applications_api: argocd.

I ensured that Argo could update the apps by updating the RBAC role as described in the official App in Any namespace doc.

In this mode, it correctly detects the Argo Application, updates it (see above), but then it fails to update the parameter in Argo, because it does not pass the namespace of the app, and Argo uses argocd by default when no namespace is specified (see logs below).

Version
0.12.2

Logs

Those logs are after configuring applications_api: argocd

argocd-image-updater logs:

time="2023-08-05T08:44:46Z" level=debug msg="found 1 from 1 tags eligible for consideration" image="rg.fr-par.scw.cloud/my-project/test-image:production"
time="2023-08-05T08:44:46Z" level=info msg="Setting new image to rg.fr-par.scw.cloud/my-project/test-image@sha256:d468cd177dcc4e0bedb94492b7e171c9478ad10486bb40f021caa02f9a2ab660" alias=test-image application=test-image image
_name=my-project/test-image image_tag=dummy registry=rg.fr-par.scw.cloud
time="2023-08-05T08:44:46Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=test-image image=rg.fr-par.scw.cloud/my-project/test-image
time="2023-08-05T08:44:46Z" level=info msg="Successfully updated image 'rg.fr-par.scw.cloud/my-project/test-image@dummy' to 'rg.fr-par.scw.cloud/my-project/test-image@sha256:d468cd177dcc4e0bedb94492b7e171c9478ad10486bb40f021caa02f
9a2ab660', but pending spec update (dry run=false)" alias=test-image application=test-image image_name=my-project/test-image image_tag=dummy registry=rg.fr-par.scw.cloud
time="2023-08-05T08:44:46Z" level=debug msg="Using commit message: "
time="2023-08-05T08:44:46Z" level=info msg="Committing 1 parameter update(s) for application test-image" application=test-image
time="2023-08-05T08:44:46Z" level=error msg="Could not update application spec: rpc error: code = PermissionDenied desc = permission denied" application=test-image
time="2023-08-05T08:44:46Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"

Note that the namespace for the app does not appear anywhere here.

argocd-server logs:

time="2023-08-05T08:42:44Z" level=info msg="received unary call /application.ApplicationService/List" grpc.method=List grpc.request.content="resourceVersion:\"\" selector:\"\" repo:\"\" " grpc.service=application.Applica
tionService grpc.start_time="2023-08-05T08:42:44Z" span.kind=server system=grpc
time="2023-08-05T08:42:44Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2023-08-05T08:42:44Z" grpc.time_ms=0.573 span.kind=
server system=grpc
time="2023-08-05T08:42:45Z" level=info msg="received unary call /application.ApplicationService/UpdateSpec" grpc.method=UpdateSpec grpc.request.content="%!v(PANIC=String method: reflect.Value.Bytes of non-byte slice)" gr
pc.service=application.ApplicationService grpc.start_time="2023-08-05T08:42:45Z" span.kind=server system=grpc
time="2023-08-05T08:42:45Z" level=warning msg="application does not exist" application=test-image namespace=argocd
time="2023-08-05T08:42:45Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = permission denied" grpc.code=PermissionDenied grpc.method=UpdateSpec grpc.se
rvice=application.ApplicationService grpc.start_time="2023-08-05T08:42:45Z" grpc.time_ms=7.134 span.kind=server system=grpc

The error is here: msg="application does not exist" application=test-image namespace=argocd (namespace is incorrect)

[JStacy86]

Is there an update to this? This is a huge problem and basically defeats the purpose of the image updater if you can't use it in any namespaces outside of where argocd is installed. The documentation should reflect this.

[NissesSenap]

This issue should now be solved through: #763

[wd]

I think #763 hasn't solved this issue. (I have tested the master branch and it looks like I'm correct)
This issue was for the problem that argo-image-updater only supports applications in the good namespace.
AIU will not process applications in other namespaces.

I think the root cause is here https://github.com/argoproj-labs/argocd-image-updater/blob/master/cmd/run.go#L113, the kubeclient defined here is restricted to the argocd namespace.

I think perhaps we can try with APPLICATIONS_API https://github.com/argoproj-labs/argocd-image-updater/blob/master/cmd/run.go#L221; this approach will use an API token to interact with ArgoCD which I think could work with apparitions in all the namespaces.

Updates: I got the API approach worked in my environment.

[jortkoopmans]

I've also encountered this issue (v0.14.0 and built from master).
Using APPLICATIONS_API works, the problem is therefore indeed isolated to the k8sclient as described by @wd .

I've added a draft PR that aims to help pinpointing the issue and working towards a solution. Since I think there are some design decisions that need to be taken, I'm looking forward to your feedback 😃 .

[jortkoopmans]

@chengfang ; Since you're aware of this issue (from your comment on another ticket), what is your view on the resolution direction?

[ppodevlabs]

any progress on the issue? this is a must for us to provide the feature to the users.