Unable to pull images when using kubelet.config.k8s.io/v1

[dntosas]

Hey team 👋

Trying to make a k8s cluster v1.27 work with Kubelet alongside with ecr-creds-helper following this guide https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/

Using IAM attached on node, both IMDSv1/v2 allowed.

This is my configuration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: CredentialProviderConfig
providers:
  - name: docker-credential-ecr-login
    matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
      - "public.ecr.aws"
    defaultCacheDuration: "12h"
    args:
    - get
    apiVersion: credentialprovider.kubelet.k8s.io/v1

Getting always this error, tried multiple configs/args etc. Any clues?

Error:

level=error msg="Error parsing the serverURL" error="parse \"https://{\\\"kind\\\":\\\"CredentialProviderRequest\\\",\\\"apiVersion\\\":\\\"credentialprovider.kubelet.k8s.io/v1beta1\\\",\\\"image\\\":\\\"XXXX.dkr.ecr.eu-west-1.amazonaws.com/XXXX\\\"}\": invalid port \":\\\"credentialprovider.kubelet.k8s.io\" after host" serverURL="{\"kind\":\"CredentialProviderRequest\",\"apiVersion\":\"credentialprovider.kubelet.k8s.io/v1beta1\",\"image\":\"XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX\"}"

Error parsing the serverURL invalid port

[dntosas]

Seems it produced from here https://github.com/awslabs/amazon-ecr-credential-helper/blob/main/ecr-login/api/client.go#L58

[MadJlzz]

I am facing the same issue. It's because the kubelet is sending a more complex request than just the registry URL.

(described here)

I am not sure how the implement this since it needs to solve both Docker credentials plugin / Kubelet credentials plugin.

I just found another implementation that looks suitable for the Kubelet in the cloud-provider repo.
https://github.com/kubernetes/cloud-provider-aws/blob/master/cmd/ecr-credential-provider/main.go

[MadJlzz]

I just tested what I mentioned above (using the ecr-credential-provider) from the cloud-provider-aws repository and I can confirm it works.

I suggest to add few lines redirecting Kubernetes users to the correct project to avoid confusion.
WDYT?

[ryparker]

@MadJlzz Testing with the cloud-provider-aws provider you suggest and still getting

pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials

what does your credential-provider-config.yaml look like?

[MadJlzz]

Something like this

{
    "providers": [
        {
            "name": "ecr-credential-provider",
            "matchImages" : [
                "<account_number>.dkr.ecr.*.amazonaws.com",
            ],
            "apiVersion": "credentialprovider.kubelet.k8s.io/v1",
            "defaultCacheDuration": "0"
        }
    ]
}

Dont forget to pass --image-credential-provider-bin-dir and --image-credential-provider-config to the kubelet.

Also, the above name property should match the name of the binary on your filesystem.

Finally, make sure you have the correct IAM permissions set on the role that makes the call.

[ryparker]

Getting the following error with k8s 1.25

no kind \"CredentialProviderConfig\" is registered for version \"kubelet.config.k8s.io/v1\" in scheme \"pkg/credentialprovider/plugin/plugin.go:56\""

Looks like I have to use v1beta1 but then I get an error from ecr-credential-provider

E0330 15:58:32.678269   60816 main.go:267] Error running credential provider plugin: group version credentialprovider.kubelet.k8s.io/v1beta1 is not supported

Only fix was to upgrade to k8s 1.29

[MadJlzz]

Indeed, I tested on a Kubernetes 1.29 but it is stable from 1.26 onward.
https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/

[dntosas]

for anyone landed in here, with same issue --> https://github.com/dntosas/ecr-credential-provider