401 error while pushing since 1.2.11

[nudgegoonies]

Since our update to lstags 1.2.11 we get 401 errors on pushing when images have changed.

I turned on tracing and verbosity with 1.2.10 and 1.2.11 and copied the relevant sections for the image/tag that fails with 1.2.11.
This is the working 1.2.10:

b6874ea|@URL: https://docker.elastic.co/v2/kibana/kibana/tags/list
b6874ea|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:09:08 GMT]
b6874ea|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
b6874ea|@HEADER: X-Content-Type-Options                   = [nosniff]
b6874ea|@HEADER: Content-Length                           = [2278]
b6874ea|@HEADER: Connection                               = [keep-alive]
b6874ea|@HEADER: Content-Type                             = [application/json; charset=utf-8]
b6874ea|--- BODY BEGIN ---
b6874ea|{"name":"kibana/kibana","tags":["5.0.0-731e78df","5.0.0-86a0b164","5.0.0-alpha5","5.0.0-beta1","5.0.0-ccd69424","5.0.0-rc1","5.0.0","5.0.1-6d75f4de","5.0.1-c068af18","5.0.1","5.0.2","5.1.1-71988514","5.1.1","5.1.2","5.2.0-11b63c8d","5.2.0-6ee3d9ba","5.2.0-a9f63967","5.2.0","5.2.1","5.2.2","5.3.0-d5b30bd7","5.3.0","5.3.1","5.3.2","5.3.3","5.4.0","5.4.1","5.4.2","5.4.3","5.5.0-e210c18b","5.5.0","5.5.1","5.5.2","5.5.3","5.6.0","5.6.1","5.6.10","5.6.11","5.6.12","5.6.13","5.6.14","5.6.15","5.6.16","5.6.2","5.6.3","5.6.4-1fdad85","5.6.4","5.6.5","5.6.6","5.6.7","5.6.8","5.6.9","6.0.0-alpha1","6.0.0-alpha2","6.0.0-beta1","6.0.0-beta2-d2816397","6.0.0-beta2","6.0.0-rc1","6.0.0-rc2","6.0.0","6.0.1","6.1.0","6.1.1","6.1.2","6.1.3","6.1.4","6.2.0","6.2.1","6.2.2","6.2.3","6.2.4","6.3.0","6.3.1","6.3.2","6.3.3-SNAPSHOT","6.4-SNAPSHOT","6.4.0-SNAPSHOT","6.4.0","6.4.1-SNAPSHOT","6.4.1","6.4.2-SNAPSHOT","6.4.2","6.4.3-SNAPSHOT","6.4.3","6.4.4-SNAPSHOT","6.5-SNAPSHOT","6.5.0-SNAPSHOT","6.5.0","6.5.1-109","6.5.1-SNAPSHOT","6.5.1","6.5.2-SNAPSHOT","6.5.2","6.5.3-SNAPSHOT","6.5.3","6.5.4-SNAPSHOT","6.5.4","6.5.5-SNAPSHOT","6.6-SNAPSHOT","6.6.0-SNAPSHOT","6.6.0","6.6.1-SNAPSHOT","6.6.1","6.6.2-SNAPSHOT","6.6.2","6.6.3-SNAPSHOT","6.7-SNAPSHOT","6.7.0-SNAPSHOT","6.7.0","6.7.1-SNAPSHOT","6.7.1","6.7.2-SNAPSHOT","6.7.2","6.7.3-SNAPSHOT","6.8-SNAPSHOT","6.8.0-SNAPSHOT","6.8.0","6.8.1-SNAPSHOT","6.8.1-e43faf64","6.8.1","6.8.2-SNAPSHOT","6.8.2","6.8.3-SNAPSHOT","6.8.3","6.8.4-SNAPSHOT","6.8.4","6.8.5-SNAPSHOT","6.8.5","6.8.6-SNAPSHOT","6.8.6","6.8.7-SNAPSHOT","6.x-SNAPSHOT","7.0-SNAPSHOT","7.0.0-SNAPSHOT","7.0.0-alpha1-SNAPSHOT","7.0.0-alpha1","7.0.0-alpha2","7.0.0-beta1","7.0.0-rc1","7.0.0-rc2","7.0.0","7.0.1-SNAPSHOT","7.0.1","7.0.2-SNAPSHOT","7.1-SNAPSHOT","7.1.0-SNAPSHOT","7.1.0","7.1.1-SNAPSHOT","7.1.1","7.1.2-SNAPSHOT","7.2-SNAPSHOT","7.2.0-SNAPSHOT","7.2.0","7.2.1-SNAPSHOT","7.2.1","7.2.2-SNAPSHOT","7.3-SNAPSHOT","7.3.0-SNAPSHOT","7.3.0","7.3.1-SNAPSHOT","7.3.1","7.3.2-SNAPSHOT","7.3.2","7.3.3-SNAPSHOT","7.4-SNAPSHOT","7.4.0-SNAPSHOT","7.4.0","7.4.1-SNAPSHOT","7.4.1","7.4.2-SNAPSHOT","7.4.2","7.4.3-SNAPSHOT","7.5-SNAPSHOT","7.5.0-SNAPSHOT","7.5.0","7.5.1-SNAPSHOT","7.5.1","7.5.2-SNAPSHOT","7.6.0-SNAPSHOT","7.x-SNAPSHOT","8.0.0-SNAPSHOT","master-SNAPSHOT"]}
b6874ea|
b6874ea|--- BODY END ---
[...]
12c6fdb|@URL: https://$ARTIFACTORY/v2/kibana/kibana/tags/list
12c6fdb|@HEADER: Connection                               = [keep-alive]
12c6fdb|@HEADER: Server                                   = [Artifactory/6.16.0]
12c6fdb|@HEADER: X-Artifactory-Id                         = [$ID]
12c6fdb|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
12c6fdb|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:09:15 GMT]
12c6fdb|@HEADER: Content-Type                             = [application/json]

This is the failing 1.2.11:

d55957a|@URL: https://docker.elastic.co/v2/kibana/kibana/tags/list
d55957a|@HEADER: Content-Type                             = [application/json; charset=utf-8]
d55957a|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:03:50 GMT]
d55957a|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
d55957a|@HEADER: X-Content-Type-Options                   = [nosniff]
d55957a|@HEADER: Content-Length                           = [2278]
d55957a|@HEADER: Connection                               = [keep-alive]
d55957a|--- BODY BEGIN ---
d55957a|{"name":"kibana/kibana","tags":["5.0.0-731e78df","5.0.0-86a0b164","5.0.0-alpha5","5.0.0-beta1","5.0.0-ccd69424","5.0.0-rc1","5.0.0","5.0.1-6d75f4de","5.0.1-c068af18","5.0.1","5.0.2","5.1.1-71988514","5.1.1","5.1.2","5.2.0-11b63c8d","5.2.0-6ee3d9ba","5.2.0-a9f63967","5.2.0","5.2.1","5.2.2","5.3.0-d5b30bd7","5.3.0","5.3.1","5.3.2","5.3.3","5.4.0","5.4.1","5.4.2","5.4.3","5.5.0-e210c18b","5.5.0","5.5.1","5.5.2","5.5.3","5.6.0","5.6.1","5.6.10","5.6.11","5.6.12","5.6.13","5.6.14","5.6.15","5.6.16","5.6.2","5.6.3","5.6.4-1fdad85","5.6.4","5.6.5","5.6.6","5.6.7","5.6.8","5.6.9","6.0.0-alpha1","6.0.0-alpha2","6.0.0-beta1","6.0.0-beta2-d2816397","6.0.0-beta2","6.0.0-rc1","6.0.0-rc2","6.0.0","6.0.1","6.1.0","6.1.1","6.1.2","6.1.3","6.1.4","6.2.0","6.2.1","6.2.2","6.2.3","6.2.4","6.3.0","6.3.1","6.3.2","6.3.3-SNAPSHOT","6.4-SNAPSHOT","6.4.0-SNAPSHOT","6.4.0","6.4.1-SNAPSHOT","6.4.1","6.4.2-SNAPSHOT","6.4.2","6.4.3-SNAPSHOT","6.4.3","6.4.4-SNAPSHOT","6.5-SNAPSHOT","6.5.0-SNAPSHOT","6.5.0","6.5.1-109","6.5.1-SNAPSHOT","6.5.1","6.5.2-SNAPSHOT","6.5.2","6.5.3-SNAPSHOT","6.5.3","6.5.4-SNAPSHOT","6.5.4","6.5.5-SNAPSHOT","6.6-SNAPSHOT","6.6.0-SNAPSHOT","6.6.0","6.6.1-SNAPSHOT","6.6.1","6.6.2-SNAPSHOT","6.6.2","6.6.3-SNAPSHOT","6.7-SNAPSHOT","6.7.0-SNAPSHOT","6.7.0","6.7.1-SNAPSHOT","6.7.1","6.7.2-SNAPSHOT","6.7.2","6.7.3-SNAPSHOT","6.8-SNAPSHOT","6.8.0-SNAPSHOT","6.8.0","6.8.1-SNAPSHOT","6.8.1-e43faf64","6.8.1","6.8.2-SNAPSHOT","6.8.2","6.8.3-SNAPSHOT","6.8.3","6.8.4-SNAPSHOT","6.8.4","6.8.5-SNAPSHOT","6.8.5","6.8.6-SNAPSHOT","6.8.6","6.8.7-SNAPSHOT","6.x-SNAPSHOT","7.0-SNAPSHOT","7.0.0-SNAPSHOT","7.0.0-alpha1-SNAPSHOT","7.0.0-alpha1","7.0.0-alpha2","7.0.0-beta1","7.0.0-rc1","7.0.0-rc2","7.0.0","7.0.1-SNAPSHOT","7.0.1","7.0.2-SNAPSHOT","7.1-SNAPSHOT","7.1.0-SNAPSHOT","7.1.0","7.1.1-SNAPSHOT","7.1.1","7.1.2-SNAPSHOT","7.2-SNAPSHOT","7.2.0-SNAPSHOT","7.2.0","7.2.1-SNAPSHOT","7.2.1","7.2.2-SNAPSHOT","7.3-SNAPSHOT","7.3.0-SNAPSHOT","7.3.0","7.3.1-SNAPSHOT","7.3.1","7.3.2-SNAPSHOT","7.3.2","7.3.3-SNAPSHOT","7.4-SNAPSHOT","7.4.0-SNAPSHOT","7.4.0","7.4.1-SNAPSHOT","7.4.1","7.4.2-SNAPSHOT","7.4.2","7.4.3-SNAPSHOT","7.5-SNAPSHOT","7.5.0-SNAPSHOT","7.5.0","7.5.1-SNAPSHOT","7.5.1","7.5.2-SNAPSHOT","7.6.0-SNAPSHOT","7.x-SNAPSHOT","8.0.0-SNAPSHOT","master-SNAPSHOT"]}
d55957a|
d55957a|--- BODY END ---

Bad response status: 401 Unauthorized >> https://$ARTIFACTORY/v2/kibana/kibana/tags/list

Because this error comes from authentication there is no useful log from Artifactory. The nginx log shows these:

[09/Jan/2020:08:03:15 +0000] "GET /artifactory/api/docker/docker-external-local/v2/kibana/kibana/tags/list HTTP/1.1" 200 119 "-" "Go-http-client/1.1"
[09/Jan/2020:08:09:56 +0000] "GET /artifactory/api/docker/docker-external-local/v2/kibana/kibana/tags/list HTTP/1.1" 401 101 "-" "Go-http-client/1.1"

I suspect this has to do with the new token handling that was merged into 1.2.11.

[ivanilves]

Thanks for this info! I demoted 1.2.11 to "pre-release", so latest would be 1.2.10 again.

Hope to have some time in nearest future to investigate this.

[nudgegoonies]

Thank you. We reverted to 1.2.10.

[ivanilves]

Please forgive me this long delay. I had a lot of involvement with sick kids and so on. 😓

I've released https://github.com/ivanilves/lstags/releases/tag/v1.2.14

So you can force BASIC auth by adding a parameter:

-B external.docker.mamdev.server.lan username:password

Still I fail to understand what was really broken since 1.2.11.

But here we just try to see if an issue could at least worked around 🤷‍♂️

[nudgegoonies]

No problem. I was ill too. Since yesterday i started homeoffice. Yes, with the parameter it works.

[nudgegoonies]

My team-lead and i discussed today about the authentication problem since 1.2.11.

It seems that lstags constructs a jwt token altough Artifactory has responded with a plain token.

Please have a look at my team-leads comments in this containerd issue:
containerd/containerd#3556 (comment)
containerd/containerd#3556 (comment)

See also these docker links:
https://docs.docker.com/registry/spec/auth/token/

jwt is an optional token method for docker registries:
https://docs.docker.com/registry/spec/auth/jwt/

[ivanilves]

Thank you very much for this input! lstags has always respected the remote registry authentication preferences and the only thing I could potentially blame here is my connection cache addition somehow messed up its decision making. Need to play more with Artifactory on my local PC to understand it... 😄

[cablespaghetti]

This also seems to be an issue on ECR. Downgrading to 1.2.10 fixes it.