update-strategy digest fails with amazon ECR api?

[diranged]

Describe the bug
I'm trying to get around #216 by using the new "digest" update strategy (#176). We're using AWS ECR as our backend and we set the appropriate tags on our image. It seems that the digest mode is set (based on the "setting dummy ..." line in the logs), but something is going wrong with populating the image digest of the latest latest image. Here are the logs (I've shared this with @jannfis on Slack privately, but I felt like I should open a bug report here):

time="2021-06-15T19:17:39Z" level=trace msg="processing app 'ourimag-basic' of type 'Helm'"
time="2021-06-15T19:17:39Z" level=debug msg="Processing application ourimag-basic"
time="2021-06-15T19:17:39Z" level=debug msg="Considering this image for update" alias=ourimag application=ourimag-basic image_name=ourimag image_tag=latest registry=1234.dkr.ecr.us-west-2.amazonaws.com
time="2021-06-15T19:17:39Z" level=debug msg="Using version constraint 'latest' when looking for a new tag" alias=ourimag application=ourimag-basic image_name=ourimag image_tag=latest registry=1234.dkr.ecr.us-west-2.amazonaws.com
time="2021-06-15T19:17:39Z" level=trace msg="found update strategy digest in argocd-image-updater.argoproj.io/ourimag.update-strategy"
time="2021-06-15T19:17:39Z" level=trace msg="No match annotation argocd-image-updater.argoproj.io/ourimag.tag-match found"
time="2021-06-15T19:17:39Z" level=trace msg="No ignore-tags annotation argocd-image-updater.argoproj.io/ourimag.ignore-tags found"
time="2021-06-15T19:17:39Z" level=info msg=/app/scripts/ecr-login.sh dir= execID=gpwGM
time="2021-06-15T19:17:40Z" level=trace msg="No secret annotation argocd-image-updater.argoproj.io/ourimag.pull-secret found"
time="2021-06-15T19:17:40Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/ourimag/tags/list"
time="2021-06-15T19:17:40Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/ourimag/tags/list?last=ukD72mdD%2FmC8b5xV3susmJzzaTgp3hKwR9nRUW1yZZ57ZeMWref7gjseb66P7uf0HAVjEXrFnb4OHhwxAKMAb9T9yvrLJ6XXFQdjH0NubJWOoldEmVa6n9tG88nfx8J3tc7EyZJ1FvlU2WW4NPTs4faqcWYF40BkztaOm6pzCyOq7v4oLog1LomLV135hrBCxa1Q6PJUVxBdcYLhYSfYKuqHl7%2Bi%2BenYmoItMvJg6LWHzrQsoxROu%2BJA5Tux2ugs%2FEpNg3dHudai59A3ejmzkIv0Q84SUJpkJ%2FGDTdq2TJWdQOc5dJxmWzmhaFoqk%2F5D24qDS8t0eYzqJ%2B1%2BtWGyBpfOkXEpm1iacahsnz%2BsZEM%3D"
time="2021-06-15T19:17:41Z" level=trace msg="List of available tags found: [main-9dc110b test-032881f test-c5b7308 main-de66320 test-1cb225e test-f89872e test-9a88600 test-59ecfc3 test-ae20a52 test-9868510 test-26e427d test-6e6e571 main-c7841fe test-7720d17 test-9393d98 test-489d70a test-cd3c3a4 test-742abf7 test-bbc136a main-435ae68 main-0da85a5 test-0903529 test-02a08b0 test-7057664 test-88a57ac test-ce56ce1 test-c5786fb test-7615772 test-1b19f1e test-3255d46 test-9a9a002 test-02aa75d test-ef32a62 test-58a43f1 test-ce2e598 test-75e7e30 test-ea1e36e main-3dc7a25 main-5e2d313 test-3dadcaf test-6f0c3c7 test-c9dc3b4 test-c3e2ae5 test-c1c4bf2 test-9725445 test-98f46f9 main-7f2db5e main-cca0372 main-8b0251a test-500ecba test-3d33ca1 test-3ccfbc4 test-8686ca9 test-79e15dc test-1076262 main-cca901d test-29c1382 test-f7bdd1d test-e362f3a main-133516f test-729799e test-a31d547 test-2ec638e test-a082842 test-03dff5d test-0a03350 test-ace4994 test-f57ccd3 test-6f04ad4 test-e97da7d test-d6c4eed test-a795505 test-3d8a188 test-cc4be9e test-d65d303 test-a53473c test-23127a3 test-13af657 main-9bfbeef test-703baaf test-077fe19 test-5c7c554 test-e8889d8 test-1e5bc27 main-39c6706 test-8f74455 test-a47aecd test-f79b015 main-6ee4c11 test-99982d8 test-fd6f2f8 test-82d93e9 test-d13c80c test-7b43fbe test-0f9ea3f test-b5e4849 test-fc25b20 test-391b18a test-f2e7049 test-9c75699 test-6918bf3 test-1226709 test-2f8a3a8 test-5ab3e81 main-faf148f test-f588f92 main-9f971e8 test-315f4c2 test-402fd6b test-62fb5da test-5706bd1 test-40b792d test-318814d test-5d79d1f test-89255e3 test-8e28a2d test-de8ddfa test-3fe1a53 test-62b4b87 test-1cbdb18 test-73d20b7 test-1672dfe test-e8e2109 test-3bcf4b2 main-986dd2e main-a13b52e test-fdd1147 test-d1dcff1 main-7024a2a test-23fd647 test-471fdd1 test-40f14d2 main-0a8b396 main-6dc596c test-9df0228 test-6c555b1 test-ff5bd78 test-58b986f test-6c55b0c test-549320f test-d0f02b2 main-d29a6bd test-b441a90 latest test-ed82bdd main-e87cab1]" alias=ourimag application=ourimag-basic image_name=ourimag image_tag=latest registry=1234.dkr.ecr.us-west-2.amazonaws.com
time="2021-06-15T19:17:41Z" level=trace msg="Finding out whether to consider latest for being updateable" image="1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest"
time="2021-06-15T19:17:41Z" level=trace msg="Finding out whether to consider main-0a8b396 for being updateable" image="1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest"
...
time="2021-06-15T19:17:41Z" level=trace msg="test-ff5bd78 did not match contraint latest" image="1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest"
time="2021-06-15T19:17:41Z" level=debug msg="found 1 from 146 tags eligible for consideration" image="1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest"
time="2021-06-15T19:17:41Z" level=trace msg="Setting dummy digest for image 1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest"
time="2021-06-15T19:17:41Z" level=info msg="Setting new image to 1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest" alias=ourimag application=ourimag-basic image_name=ourimag image_tag=dummy registry=1234.dkr.ecr.us-west-2.amazonaws.com
time="2021-06-15T19:17:41Z" level=debug msg="target parameters: image-spec= image-name=ourimag.image.repository, image-tag=ourimag.image.tag" application=ourimag-basic image=1234.dkr.ecr.us-west-2.amazonaws.com/ourimag
time="2021-06-15T19:17:41Z" level=info msg="Successfully updated image '1234.dkr.ecr.us-west-2.amazonaws.com/ourimag@dummy' to '1234.dkr.ecr.us-west-2.amazonaws.com/ourimag:latest', but pending spec update (dry run=true)" alias=ourimag application=ourimag-basic image_name=ourimag image_tag=dummy registry=1234.dkr.ecr.us-west-2.amazonaws.com
time="2021-06-15T19:17:41Z" level=debug msg="Using commit message: "
time="2021-06-15T19:17:41Z" level=info msg="Dry run - not commiting 0 changes to application" application=ourimag-basic
time="2021-06-15T19:17:41Z" level=info msg="Finished cache warm-up, pre-loaded 0 meta data entries from 6 registries"

To Reproduce
Set up an AWS ECR repository and point the Image Updater to it (see #112 (comment)). Then set your Application annotations up like this:

Annotations:  argocd-image-updater.argoproj.io/myapp.helm.image-name: myapp.image.repository
              argocd-image-updater.argoproj.io/myapp.helm.image-tag: myapp.image.tag
              argocd-image-updater.argoproj.io/myapp.update-strategy: digest
              argocd-image-updater.argoproj.io/image-list: featurestore=1234.dkr.ecr.us-west-2.amazonaws.com/myorg/myapp:latest

Expected behavior
As soon as it discovers that latest is a valid tag that matches our image-list, I expect it to update the Application with the sha256:... hash of that tag. Instead it is setting it to latest which does not trigger a new deploy.

Version
Latest master build as of 1d19e6a

[diranged]

@jannfis Ok so it turns out this works fine. The problem is that on the repo I was testing with, no one was uploading a latest tag. So I think that the only real bug here is that the image updater just moves on to a different behavior if the latest tag simply doesn't exist, rather than clearly logging out that it cannot find a hash value for that tag.

[jannfis]

Thanks for the follow-up on this @diranged. So if I understood you correctly, the bug is like:

• Have update strategy digest
• Chose to use digest for images tagged e.g. latest
• If image updater doesn't find the tag latest in registry (because it does not exist), it will set the actual tag to latest in the update cycle, instead of just leaving it alone

Is that a correct assumption? If yes, it's definitely a bug :)

[diranged]

Thanks for the follow-up on this @diranged. So if I understood you correctly, the bug is like:

• Have update strategy digest
• Chose to use digest for images tagged e.g. latest
• If image updater doesn't find the tag latest in registry (because it does not exist), it will set the actual tag to latest in the update cycle, instead of just leaving it alone

Is that a correct assumption? If yes, it's definitely a bug :)

Yup that's basically what happened.. confused the hell out of me! :)

[jannfis]

Thank you @diranged ! I could successfully reproduce the bug, a fix is underway.