Unable to nerdctl push container after nerdctl login when using pass credsStore on Linux

[thehejik]

Actual Behavior

I'm unable to push a container image by using nerdctl push thehejik/busybox:latest after deleting default docker's ~/.docker/config.json and providing my personal dockerhub credentials. Docker/moby RD backend works normally.

Rancher-desktop version 1.4.1-373-g9051f530 built from main branch installed on openSUSE 15.3 and Ubuntu 20.04.

Steps to Reproduce

• Install RD
• delete ~/.docker/config.json (if exists)
• run RD and initialize with default values
• setup pass according to https://docs.rancherdesktop.io/getting-started/installation#pass-setup:

$ gpg --generate-key # provide full name, e-mail and confirm, do not provide a passphrase
> 2E1C8E4D37F7D82EF02176BC5B300A52EEE63CAB
$ pass init 2E1C8E4D37F7D82EF02176BC5B300A52EEE63CAB
mkdir: created directory '/home/thehejik/.password-store/'
Password store initialized for 2E1C8E4D37F7D82EF02176BC5B300A52EEE63CAB
$ pass list
Password Store
└── docker-credential-helpers
    └── aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv
        └── thehejik

• run nerdctl login and provide your private DockerHub credentials (wait for Login Succeeded)
• pull some image eg by. nerdctl pull busybox
• re-tag the pulled image by nerdctl <id_of_busybox_image> thehejik/busybox:latest (replace thehejik by your docker username)
• try to push the retagged image nerdctl push thehejik/busybox:latest

Result

The busybox image is not pushed with Error message FATA[0002] server message: insufficient_scope: authorization failed:

$ nerdctl push thehejik/busybox:latest
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:c12b4701670111b6d3518ab989138dbf9b284d464982ceaa21128e280242e0f9) 
index-sha256:c12b4701670111b6d3518ab989138dbf9b284d464982ceaa21128e280242e0f9:    waiting        |--------------------------------------| 
manifest-sha256:dcdf379c574e1773d703f0c0d56d67594e7a91d6b84d11ff46799f60fb081c52: waiting        |--------------------------------------| 
config-sha256:62aedd01bd8520c43d06b09f7a0f67ba9720bdc04631a8242c65ea995f3ecac8:   waiting        |--------------------------------------| 
elapsed: 1.8 s                                                                    total:   0.0 B (0.0 B/s)                                         
FATA[0002] server message: insufficient_scope: authorization failed

Expected Behavior

The image should be pushed to DockerHub as with Docker.

Additional Information

Works ok with docker/moby RD backend. Also docker installed on the system is working fine.

Rancher Desktop Version

1.4.1-main

Rancher Desktop K8s Version

any

Which container runtime are you using?

containerd (nerdctl)

What operating system are you using?

Ubuntu

Operating System / Build Version

Ubuntu 20.04, openSUSE 15.3

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

deb

Windows User Only

No response

[jandubois]

FYI, this works fine for me on macOS (I know the ticket is about Linux; just providing a datapoint).

[rak-phillip]

@jandubois able to repro in openSUSE Tumbleweed with the steps outlined above

[jandubois]

This fails on macOS as well when using the docker-credential-none helper. The reason seems to be that the none-helper is too strict in comparing server URLs:

Using docker-credential-osxkeychain:

$ docker-credential-osxkeychain list | jq -r '.["https://index.docker.io/v1/"]'
jandubois

$ echo https://index.docker.io/v1/ | docker-credential-osxkeychain get | jq -r .ServerURL
https://index.docker.io/v1/

$ echo https://index.docker.io/v1/ | docker-credential-osxkeychain get | jq -r .ServerURL
https://index.docker.io/v1/

$ echo https://index.docker.io | docker-credential-osxkeychain get | jq -r .ServerURL
https://index.docker.io

Compare with docker-credential-none:

$ docker-credential-none list | jq -r '.["https://index.docker.io/v1/"]'
jandubois

$ echo https://index.docker.io/v1/ | docker-credential-none get | jq -r .ServerURL
https://index.docker.io/v1/

$ echo https://index.docker.io | docker-credential-none get | jq -r .ServerURL
parse error: Invalid numeric literal at line 1, column 12

$ echo index.docker.io | docker-credential-none get | jq -r .ServerURL
parse error: Invalid numeric literal at line 1, column 12

I found that nerdctl push is calling get on index.docker.io and not the full ServerURL. So I think it never worked with the pass helper, and maybe can be considered a bug in nerdctl.

However, this may still be a regression against Rancher Desktop 1.3.x, which didn't configure a credentials helper inside the VM at all, so credentials would just be stored in /root/.docker/config.json, and therefore just worked. I'm going to verify this.

[jandubois]

Yes, this is a regression against 1.3.0 (because it didn't configure the credential helpers), but has been broken since 1.4.1:

$ nerdctl login
Enter Username: jandubois
Enter Password:
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

$ nerdctl pull busybox
docker.io/library/busybox:latest:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:dcdf379c574e1773d703f0c0d56d67594e7a91d6b84d11ff46799f60fb081c52: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:62aedd01bd8520c43d06b09f7a0f67ba9720bdc04631a8242c65ea995f3ecac8:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:19d511225f94f9b5cbf3836eb02b5273c01b95da50735742560e3e45b8c8bfcc:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.4 s                                                                    total:  758.8  (316.0 KiB/s)

$ nerdctl tag busybox jandubois/busybox:none3

$ nerdctl push jandubois/busybox:none3
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:c12b4701670111b6d3518ab989138dbf9b284d464982ceaa21128e280242e0f9)
index-sha256:c12b4701670111b6d3518ab989138dbf9b284d464982ceaa21128e280242e0f9:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:dcdf379c574e1773d703f0c0d56d67594e7a91d6b84d11ff46799f60fb081c52: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:62aedd01bd8520c43d06b09f7a0f67ba9720bdc04631a8242c65ea995f3ecac8:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.4 s                                                                    total:  2.2 Ki (1.6 KiB/s)

[jandubois]

While we can update the none helper to support pure hostname based queries, that doesn't actually help us with the real issue, which is the pass helper.

We could create a fork and make it more forgiving in what arguments it accepts, but it would be better if this was fixed in nerdctl instead. I've file containerd/nerdctl#1263 for this.

If nerdctl gets fixed before the next Rancher Desktop patch release, please make sure to update nerdctl in both the alpine-lima ISO and the WSL distro. Ask @AkihiroSuda to make a new alpine-lima release!

[jandubois]

I've created a PR against nerdctl for this: containerd/nerdctl#1270

We'll need new releases for the WSL distro and Alpine Lima ISO with an update to nerdctl 0.22.1 to get the fix.

[AkihiroSuda]

Fixed in nerdctl v0.22.2 (v0.22.1 had a regression on Fedora and Arch Linux) / alpine-lima v0.2.20 / Lima v0.11.3

[mook-as]

I seem to have issues with nerdctl 0.22.2 on macOS; I'll continue the investigation later, but currently it looks like:

$ nerdctl pull mookas/go-env
FATA[0000] expected ac.ServerAddress ("") to be "https://index.docker.io/v1/" 
$ nerdctl version
Client:
 Version:	v0.22.2
 OS/Arch:	linux/amd64
 Git commit:	2899222cb0715f1e5ffe356d10c3439ee8ee3ba4
 buildctl:
  Version:	v0.10.3
  GitCommit:	c8d25d9a103b70dc300a4fd55e7e576472284e31

Server:
 containerd:
  Version:	v1.6.6
  GitCommit:	10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:	1.1.2
  GitCommit:	a916309fff0f838eb94e928713dbc3c0d0ac7aa4

Confusingly, this works on Windows, where we should be using the same version (0.22.2)… so it's probably something silly I'm doing.

[mook-as]

Hmm, turns out my issue was just that we needed to login (but the error was really bad).