Malformed response when requesting EKS token from gov-cloud (affects `>=2.10.4`)

[timfallmk]

Describe the bug

After updating to cli v 2.10.4 (and continuing in 2.11.0) we get the following response from our clusters running in Gov-Cloud

kubectl get pods
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

We get the same response from every kubectl request.

Narrowed the problem down to the authentication step callout to aws eks get-token. With debug logs on, error occurs after the following lines:

host:sts.us-gov-west-1.amazonaws.com
x-k8s-aws-id:farpoint

host;x-k8s-aws-id
<earlier call stack>
<snip>
2023-03-03 13:43:15,726 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20230303T214315Z
20230303/us-gov-west-1/sts/aws4_request
<snip>
2023-03-03 13:43:15,726 - MainThread - botocore.auth - DEBUG - Signature:
<snip>

A few key details:

• Region: us-gov-west-1
• kubectl version:

Client Version: v1.25.6
Kustomize Version: v4.5.7
Server Version: v1.25.6-eks-48e63af

• Clusters are EKS running 1.25 and provisioned with eksctl
• We use aws sso for account auth and token generation

Confirmed a few things:

• Does not happen when using fixed IAM credentials, either via aws or aws-iam-authenticator
• Does not happen for cli <=2.10.3

I can't seem to get any more debug logs so I can't dig further, but feels like a malformed response somewhere? Naked aws eks get-token seems to return expected results.

Expected Behavior

kubectl command returns data

Current Behavior

kubectl get pods
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

Happens for all subcommands checked (that interact with the server)

Reproduction Steps

Needs:

• EKS cluster 1.25 in Gov-Cloud region
• AWS SSO setup and used as login method

Doesn't work:

asdf shell awscli 2.10.4/2.11.0/latest
kubectl X
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

Does work:

asdf shell awscli 2.10.3 (or before)
kubectl X
<expected output>

Possible Solution

Smells like a malformed HTML response is trying to get unmarshalled (just to my 👃 )
Downgrading to 2.10.3 or earlier is a viable workaround for now

Additional Information/Context

AWS SSO connected to Azure AD IdP

CLI version used

2.10.4+

Environment details (OS name and version, etc.)

macOS 13.2.1, kubectl and awscli via homebrew

[psemeniuk]

It seems that after 2.10.3 awscli relies on configured output format. Before it was json despite configured format:

aws --version; aws --region us-east-1 eks get-token --cluster-name REDACTED --role REDACTED aws-cli/2.10.3 Python/3.11.2 Darwin/21.6.0 source/arm64 prompt/off {"kind": "ExecCredential", "apiVersion": "client.authentication.k8s.io/v1beta1", "spec": {}, "status": {"expirationTimestamp": "2023-03-06T12:25:46Z", "token": "REDACTED"}}
vs
aws --version; aws --region us-east-1 eks get-token --cluster-name REDACTED --role REDACTED aws-cli/2.11.0 Python/3.11.2 Darwin/21.6.0 source/arm64 prompt/off client.authentication.k8s.io/v1beta1 ExecCredential STATUS 2023-03-06T12:28:05Z REDACTED

(i have configured text)

as a workaround you can set AWS_DEFAULT_OUTPUT env variable ( to json:

export AWS_DEFAULT_OUTPUT=json; aws --version; aws --region us-east-1 eks get-token --cluster-name REDACTED --role REDACTEED aws-cli/2.11.0 Python/3.11.2 Darwin/21.6.0 source/arm64 prompt/off { "kind": "ExecCredential", "apiVersion": "client.authentication.k8s.io/v1beta1", "spec": {}, "status": { "expirationTimestamp": "2023-03-06T12:29:08Z", "token": "REDACTED" } }

You can embed this variable by editing kubeconfig file.

[tim-finnigan]

This issue should be fixed via the PR linked above (#7726). Those changes will be available in the next CLI release.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[timfallmk]

Correct me if I'm wrong, but this would not fix the issue for current users (unless they (re)generated kubeconfig entry using the update-kubeconfig command.

[psemeniuk]

As i see from PR description, it just restores previous behaviour (before 2.10.4), so it should work out of the box.